evals-outputs/GAUNTLET.md

# gauntlet results

These are are this model's output results on my "summarization gauntlet". You can find more info about that [here on my dropbox for it](https://www.dropbox.com/sh/axu1xlscrrexy55/AADAm01-4Zs3POyHQrgbDAsda?dl=0) or at [this dataset](https://huggingface.co/datasets/pszemraj/summcomparer-gauntlet-v0p1).

- if you aren't familiar with it, one thing to note is some of the docs **purposefully** are "messy"/have spelling errors etc.

parameters

```json
{
  "model_name_or_path": "pszemraj/led-base-book-summary",
  "use_cuda": true,
  "token_batch_length": 16384,
  "batch_stride": 16,
  "max_length_ratio": 0.25,
  "load_in_8bit": false,
  "compile_model": true,
  "optimum_onnx": false,
  "device": "cuda",
  "inference_params": {
    "min_length": 8,
    "max_length": 4096,
    "no_repeat_ngram_size": 3,
    "encoder_no_repeat_ngram_size": 4,
    "repetition_penalty": 2.5,
    "num_beams": 10,
    "num_beam_groups": 1,
    "length_penalty": 1.0,
    "early_stopping": true,
    "do_sample": false
  },
  "textsum_version": "0.2.0"
}
```

- Created: `2023-11-28T20:10:42.701696`

## ASR-whisper-rpunctuated_Noam Chomsky, Fundam_1669853561_0_part1_summary

Well, he's going to talk a little bit about the basic issues of linguistics in two separate talks. First, he wants to talk about how we're trying to understand language. Specifically, what do we mean by "real" explanations? Theoretically, there are three main things: 1) an authentic explanation; 2) a genuine solution; and 3) a kind of combinatorial approach to understanding language. Brain scientists have been working on this for quite some time, but they haven't yet come up with a really good explanation. In the early 1900s, people like Otto Jesperson, Leonard Bloomfield, Robert Ademian, and others all tried to figure out how the structure of language comes into existence in their minds. They all thought that it was just a bunch of random bits of information floating around in their brains. But now science is starting to use machine learning as a way to understand languages.

---

Section Scores for ASR-whisper-rpunctuated_Noam Chomsky, Fundam_1669853561_0_part1_summary:

 - -0.9902

---

## ASR-whisper-rpunctuated_Noam Chomsky, Fundam_1669853631_0_part2_summary

Adam explains the concept merge, which he calls a "superior version of merge" because it has only two elements: an external and an internal merge. In order to explain how one can combine multiple objects, however, one must first have access to all of them. This is particularly important in the context of machine learning, where there are so many different kinds of concurrent operations that it becomes very difficult to understand what each operation accomplishes.
Pisa argues that the problem is not just a problem with the structure of the examples but also with the problems inherent in the field. The problems are "the kinds of problems" that arise when one tries to give an adequate explanation of what's happening in the real world. For example, if you create a new object and then merge it with another object, you don't get exactly what you wanted; instead, you end up using the same paradigm as all the examples before. In other words, there isn't a single solution to the problem because there aren't enough problems to solve in a principled way.

---

Section Scores for ASR-whisper-rpunctuated_Noam Chomsky, Fundam_1669853631_0_part2_summary:

 - -0.9733

 - -0.9339

---

## ASRnlp_law_lecture_week_1_v_2_c_transcription_1_summary

This is the first chapter of a new course called "Natural Language Processing", which focuses on natural language processing. The course will focus on using machine learning and artificial intelligence to understand legal and social science applications. It will also take place in person rather than in a web-based environment. There will be no overtaken by artificial intelligence or machine learning, but there will certainly be hybrid learning and hybrid learning.

---

Section Scores for ASRnlp_law_lecture_week_1_v_2_c_transcription_1_summary:

 - -0.9214

---

## ASRnlp_law_lecture_week_2_v_2_c_transcription_2_summary

Gulliver explains the changes that have been made to the course in order to better prepare students for the next generation of jobs. For example, there is now more room available for people to join remotely. There are also new questions asking if homework emissions can be counted as gas or non-acronym courses and how they should be counted. The first question is whether homework emissions must be copy-and-pasted from the note; this is important because it will allow students to know what's going on in their own notebooks at the time of joining. Next up is a survey of corporate and dictionary methods. This paper focuses on using machine learning to identify which documents are most likely to be classified as being classified as having "minor" or "high dimensionality". It also looks at ways to reduce the amount of time that companies spend on these documents.

---

Section Scores for ASRnlp_law_lecture_week_2_v_2_c_transcription_2_summary:

 - -1.0658

---

## ASRnlp_law_lecture_week_3_part_1_v_2_c_transcription_3_summary

The hybrid learning experts discuss immigrants and how they can reduce the number of words that are used in the English language. They begin with a discussion about immigrants and then move to a discussion of "reconcilable differences" between the different types of immigrants. This paper focuses on three methods for achieving this goal. The first method is called tariff similarity; the second method is Energetic Analysis. The third method is termed elastic search. The fourth method is time-series analysis. It uses machine learning to predict the frequency of words, phrases, and events in order to identify which words are most likely to be associated with default.

---

Section Scores for ASRnlp_law_lecture_week_3_part_1_v_2_c_transcription_3_summary:

 - -0.9206

---

## Emie_dissertation_cleansed_summary

Szemraj's dissertation is a "Master of Philosophy" in film and screen studies. It focuses on three subjects: urban space, materiality, and movement in post-war American and British films noir; act of violence; and the man between. Act of Violence takes place in Los Angeles during the aftermath of the German invasion of the United States in World War II. The focus of the film is on how the characters move through the city as if they are moving from physical reality to metaphysically real reality. In Act 2, for example, Enley walks through Bunker Hill with his dog, Parkson. When he reaches the train tracks, he sees that they are part of the wreckage of the Berlin Airlift.
The Man Between and Act of Violence are also notable for their use of "physical correspondences" to tell the story of Susanne's captivity. Ivo attempts to persuade Susanne to go to the opera that evening, but Susanne misinterprets him as a renegade accomplice in a plot to destroy her identity. When Ivo finally succeeds in capturing the attention of the border patrol officers, he runs away into the snow. The sequence culminates with Ivo's pursuit of an errand boy who has hidden behind a laundry truck. In this sequence, Reed uses "material reality" to heighten and reveal how much Susanne is afraid of Ivo. As the action moves forward, it becomes increasingly difficult for Ivo to escape the physical reality of Berlin. Both Enley and Parkon move through suburbia or hotel lobbies in order to shed their past and present identities; however, they cannot escape the sense of alienation felt by the world around them. Thus, both films emphasize the importance of "reconciling material reality" with "artificiality."

---

Section Scores for Emie_dissertation_cleansed_summary:

 - -0.9783

 - -0.9207

---

## OCR_ML4HLecture02image__summary

This paper focuses on machine learning for medical image analysis. Machine Learning is used to predict the volume and length of abdominal organs and focal points in order to predict disease, stroke, and neurodegeneration. Dr. Barlow demonstrates how machine learning can outperform human-to-machine translation in predicting disease outcomes.

---

Section Scores for OCR_ML4HLecture02image__summary:

 - -0.8923

---

## OCR_ML4HLecture04RepresentationLearning.pptx__summary

Next, we look at how machine learning can improve the accuracy of human-caused disease prediction using time series. we use sequence-based representation learning to predict disease outcomes using real-world medical time series and compare it favorably with competitor approaches. Next, we examine how "reconcoder and forecaster" models can outperform each other in predicting disease outcomes.

---

Section Scores for OCR_ML4HLecture04RepresentationLearning.pptx__summary:

 - -0.8706

---

## OCR_ML4HLecture05-NLP.pptx__summary

Artificial Intelligence (AI) and Natural Language Processing (NLP) we examine the use of artificial intelligence to predict disease-causing behavior in real-time. Specifically, we look at how NLP tasks can be scaled down based on the frequency of occurrences of certain terms in the document. For example, if you're trying to figure out which word is most likely to show up in a given sentence, you'll find that it's actually showing up in the next sentence. The same thing happens with speech recognition.

---

Section Scores for OCR_ML4HLecture05-NLP.pptx__summary:

 - -0.9732

---

## OCR_PAPER_Hong et al. - 2022 - CogVideo Large-scale Pretraining for Text-to-Video Generation via Transformers-annotated__summary

Cog' Video, a 9B-rate transformer trained to automatically predict the future frames of text in Chinese, uses Cog' video as an example. The model outperforms its peers in both machine and human evaluation. the Tsinghua University Department of Electrical and Computer Engineering (IBEA) conducts a detailed assessment of the performance of these transformers. It is shown that they outperform all publicly available transformers when compared to other types of transformers used for text generation. Next, the team presents Cog 'Video, which is the world's first fully-fledged pretrained neural network capable of predicting future frames from real-time information. To better understand the temporal relationships between images and text, the task is further advanced by using artificial intelligence techniques such as transcoding and image generation.

---

Section Scores for OCR_PAPER_Hong et al. - 2022 - CogVideo Large-scale Pretraining for Text-to-Video Generation via Transformers-annotated__summary:

 - -1.1301

---

## OCR_PAPER_Kandpal, Nieto, Jin - 2022 - Music Enhancement via Image Translation and Vocoding-annotated__summary

Kandpal explains how he and his co-workers can improve the sound quality of music by using a combination of machine learning, maelstrom modeling, and vocoding. In particular, they demonstrate that their approach outperforms all other methods currently used to improve music quality. For example, they train a MelzMel, Diffwave, and Mel2Mel models on top of each other in order to optimally compare different levels of musical quality. The results are even better than those obtained from purely independent training.

---

Section Scores for OCR_PAPER_Kandpal, Nieto, Jin - 2022 - Music Enhancement via Image Translation and Vocoding-annotated__summary:

 - -0.9349

---

## OCR_PAPER_dall-e-2-annotated__summary

The next step in this study is to develop contrastive image generation using CLIP. To do so, the team uses diffusion models and autoregressive models. In order to achieve these results, they train multiple decoders to produce multiple images of the same image. They then use diffusion models to predict which image will be used for image generation. The final result is a high-quality dataset with over 1,000 samples from production versions of the paper.

---

Section Scores for OCR_PAPER_dall-e-2-annotated__summary:

 - -0.8975

---

## The Most Dangerous Game--Richard Connell_summary

Rainsford swears that they have reached a large island, which is rumored to be inhabited by cannibals. When Rainsford reaches the island, he feels a sudden chill and a sense of uneasiness. The ship's captain, Captain Nielsen, had warned him that the island was "an evil place" and asked him not to feel anything. He also remembers seeing an old Swede who had told him that sailors should not feel anything because the air about them was poisonous. After swimming for a few minutes in the dark, RainsfORD hears shots off in the distance. He swears to go back to sleep. On his way home, he stops at a dinner party where General Zaroff invites him to dine with him. During the course of their conversation, general Zaroff reveals that he has found a new species of deer--the Cossack--and plans to kill him if he does not find him by 3:00 p.m. the next day.

---

Section Scores for The Most Dangerous Game--Richard Connell_summary:

 - -1.0244

---

## gpt_peter_testing_group_exemplars_summary

In order to prepare for the hackathon that is to take place on Thursday evenings, Peter and his co-workers decide to create a meme about Cheesy Poofs, an American made product banned by the U.S. government because of its graphic content. The mothers of the boys ban the sale of the American-made cheese poofs in order to discourage the boys from buying them. They also issue an ultimatum to the boys regarding what they should do with their money if they want to make a profit. In order to avoid being labeled as a "hard worker," Peter changes the form of the application so that it conforms to the needs of the class. He also adds more soldiers to respond to Paul's new proposal. Lastly, the group decides to ask each other questions about their day's activities, their hobbies, and their belief in Solipsism. At the end of their training, they will discuss how they can improve the algorithm for saving the world.

---

Section Scores for gpt_peter_testing_group_exemplars_summary:

 - -0.9597

---

## navy seals copy pasta_summary

I will wipe you out with precision, mark my words. I will show you everything from the navy seals to the entire US marine corps. I will use your pathetic little life as an opportunity to wipe it all out with "unleasurable" vengeance. You're fucking dead.

---

Section Scores for navy seals copy pasta_summary:

 - -1.0145

---

## script_findingnemo_summary

FINDING Nemo: The First Day of School This is a work in progress transcript of the first part of the movie. It's not 100% accurate, and some mistakes may be made, but it's open for corrections if you have any. Marlin says he likes the neighborhood, but he doesn't want to name all of his kids after a clown. He wants to name them all "Marlin Jr." and "Coral Jr.". Brain Snack: A clownfish is a fish that swims from one place to another. In this scene, the clownfish are introduced as Dory and Bekker. They're trying to figure out how to get past the jellyfish when they run into each other on the way to meet their new families. When they finally find their families, though, they can't communicate with each other because they don't recognize each other. Everyone runs off to find their parents.
Nemo's dad, Gulliver, shows up with the news that Nemo has taken on three giant sharks. Brain Snack: Sharkbait isn't just looking for you, it's also predicting that Darla will arrive in Sydney sometime in the next 48 hours or so. In other words, they don't have much time to clean up before Darla arrives--they only have a little over 48 hours until Darla is due to arrive. The boys scramble to find their exit buddy, Squirt, who'll give them some tips on how to get out of this sticky situation. They decide to play a game of hide-and-seek in order to figure out where Dory is and what he's doing. And guess what? There's a fish floating in the water. It's called the AquaScum 2,000, which is basically a two-hundred-thousand-year-old biodegradable fish tank. Everyone freaks out except for Dory, who tries to talk to the fish by pretending to be a pelican. But Dory convinces him to try talking to Nemo instead. At the end of the chapter, Dory runs off to find his son.

---

Section Scores for script_findingnemo_summary:

 - -1.0351

 - -0.9937

---

## script_frozendisney_summary

In this short scene, the ice men work on a frozen lake. They sing a song about love and fear as they push their ice blocks through the water. An ice floe approaches them threateningly, but they fight it off. The workers pile onto a horsedrawn ice sled and head off into the Northern Lights. At night, the girls sleep in their beds. In the morning, they wake up to find Anna asleep on their bed. She has been injured by a piece of ice that accidentally knocks her over. When she wakes up, she realizes that there is no way for her to get out of the ice. A group of people from Arendelle gathers to watch the coronation of the king and queen. J. Lee introduces his newest addition to the party: Grand Pabbie. He changes all of Anna's magical memories to mundane ones like snowflakes and ice cream. As he instructs the reindeer how to control their icy hands, Anna skips down a snowy slope and lands on an ice wall. On top of her, she runs into a giant snowstorm. It freezes her whole body before she can get back up. During the celebration, Bjorn and Jaquenetta are introduced.
On the way to the ice palace, Olaf tries to kiss Anna but accidentally slams his head against a puddle of ice. He falls unconscious. Bjorn and Jaquen decide to go to the love experts who will help them find Anna. On the way home, they run into J. Lee and Olaf at an ice palace. They try to get away, but when they do, they are caught in a huge snowball-jacking. Everyone runs away except for Bjorn, who manages to save Anna's life. In the end, Bjorn is sentenced to death.
As the storm continues, Anna and Olaf rush down the hall in a panic. The walls of the building crack under the force of the storm, preventing them from escaping. When they reach the top of the wall, however, they are struck by ice spikes that block their path. They fall to the ground as a giant snowball. During this brief scene, it is revealed that Anna's sister is dead because of her. It is also revealed that Kristoff has risked his life to rescue Anna. As he runs towards her, he accidentally hits her with his sword. He falls overboard. Kai, the dockmaster, tells the guards that Arendelle will no longer conduct any dealings with HANS. The guards take him off the ship and send him back to Weselton. On the way home, Anna pulls a blindfold over her face and embraces Sven. She kisses him on the lips. At the end of the scene, Olaf skates through the crowd and helps Anna into the sleigh.

---

Section Scores for script_frozendisney_summary:

 - -1.0835

 - -1.0709

 - -0.9543

---

## script_strangersonatrain_summary

Bruno Haines, a young man in expensive clothes, and Anthony, a troubled young man, appear on the train platform. They are greeted by Bruno, who apologizes for his over-the-top smoking and remarks that he is too busy to attend the doubles match between Evans and Faraday. The conversation then moves to news of people divorcing or getting remarried. In this brief scene, they are interrupted by the arrival of a police car. As they enter the main part of the city, they witness the arrest of two men whom they believe have plotted to murder one another. At the same time, they also witness the arrests of three other men accused of plotting against them.
At the police station, Barbra tells Guy that she saw him on the train from New York City to Washington, D. C., and that he should be able to tell them where he was at nine:00 a.m. That night, Guy goes to the office of Captain Turley, who is supposed to be there. When he enters, he finds Professor Collins waiting behind him. He explains that he has been able to identify the man who tried to murder Mrs. Cunningham in the book The Murder of Mr. Metcalf by using his own name. At the same time, Haines notes that one of the Burtons' friends, Antony, has helped mastermind the murder of Dr. Calf. As they are leaving the apartment, Bertha notices that Guy is staring at her. She realizes that he is trying to talk to her about the murder and decides to leave the apartment for a few days so that she can figure out how to get rid of him. While she is gone, Anne overhears a conversation between Bruno and Miss Darville.
Bruno and Anne are in the midst of a conversation about Guy. He tells his mother that Guy should not have sent her on an errand like this, because Guy is irresponsible and will do anything for Lady Macbeth's reputation. When Anne gets up to leave, she sees Guy's lighter in her pocket. The police need something to prove that he was at the murder scene--something to prove they were at the crime scene that night. As soon as the game begins, Guy and Reynolds win the first set with a flourish. At the end of the third set, though, Guy smashes one of the horses' heads against the wall. In the fourth set, Guy kills another horseman who tries to get underneath the top of the second horse. A bystander catches a glimpse of the two men getting into a fight over a broken wheelbarrow. An onlooker rushes in to see what has happened. It turns out that Haines beat the man who tried to steal his lighter during the tournament. After the game, Ganymede and Hennessy take their cab back to the station.

---

Section Scores for script_strangersonatrain_summary:

 - -1.0462

 - -1.0986

 - -1.1284

---

## script_sunsetblvd._summary

On March 21, 1949, the Los Angeles Daily News publishes a picture of Sunset Boulevard as it was before the shooting began. Two men enter the scene and give Gillis a business card to use to get the car keys. They explain that they need to jack up the car and haul it away because they have a court order. The car is a 1946 Plymouth convertible with two shots in its back and one in its stomach. It has been loaned to a man named Sheldrake who lives next door to him. He had lent the car to a guy named Bill Demarest who took it to Palm Springs for his health. At the same time, he has borrowed three hundred dollars to pay for another car. When the men leave, Gillis' VOICE is heard on the other end of the room -- Norma's -- asking him if he can borrow some money to write a piece about her. He does not want to do anything more than talk to her. In fact, he tells her that he will send her a copy of Salome.
Gillis is embarrassed at the way Norma treats him. He tries to tell her that he doesn't want her to love him, but she tells him that he's just trying to get her to think of someone else. Gillis goes back to his apartment and finds a box with two bars of music on it. When he opens the box, there's a note saying "To Joe from Northma," which says "Mad About the Boy." The next day, Gillis goes off to New Year's Eve party at Artie Green's apartment. At the end of the night, he calls in to see how things are going. It turns out that Artie has been working on a script for a play called Norma's Salome. Afterward, they head over to De Mille's office to have a chat. There's a knock on the door as soon as they get there. In the meantime, we learn that Norma has had an affair with a guy named Gordon Cole. We also learn that de Mille wants to hire one of his assistants to help him finish his script.
Gillis tells Betty that Artie has sent her a telegram asking her to come to Arizona with him. It only costs two dollars to get married there, so it would be nice for them to have a honeymoon together. As they are finishing the story, Norma's bedroom doorbell rings and Gillis comes in to talk to her. He tries to give her some more clues but she is too scared to do so. She shoots both men in rapid succession and he falls into the pool. At the end of the scene, DeMille rushes out to save the life of Norma.

---

Section Scores for script_sunsetblvd._summary:

 - -1.0416

 - -1.0423

 - -0.8555

---