rishdotblog commited on
Commit
d5685ac
1 Parent(s): acd7307

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +62 -0
README.md ADDED
@@ -0,0 +1,62 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Defog SQLCoder
2
+ Defog's SQLCoder is a state-of-the-art LLM for converting natural language questions to SQL queries.
3
+
4
+ [Interactive Demo](https://defog.ai/sqlcoder-demo/) | [🤗 HF Repo](https://huggingface.co/defog/sqlcoder2) | [♾️ Colab](https://colab.research.google.com/drive/1z4rmOEiFkxkMiecAWeTUlPl0OmKgfEu7?usp=sharing) | [🐦 Twitter](https://twitter.com/defogdata)
5
+
6
+ ## TL;DR
7
+ SQLCoder-7B is a 7B parameter model that outperforms `gpt-3.5-turbo` for natural language to SQL generation tasks on our [sql-eval](https://github.com/defog-ai/sql-eval) framework, and significantly outperforms all popular open-source models. When fine-tuned on a given schema, it also outperforms `gpt-4`
8
+
9
+ SQLCoder is fine-tuned on a base StarCoder model.
10
+
11
+ ## Results on novel datasets not seen in training
12
+ | model | perc_correct |
13
+ |-|-|
14
+ | gpt4-2023-10-04 | 82.0 |
15
+ | defog-sqlcoder2 | 74.5 |
16
+ | gpt4-2023-08-28 | 74.0 |
17
+ | defog-sqlcoder-7b | 71.0 |
18
+ | gpt-3.5-2023-10-04 | 66.0 |
19
+ | claude-2 | 64.5 |
20
+ | gpt-3.5-2023-08-28 | 61.0 |
21
+ | claude_instant_1 | 61.0 |
22
+ | text-davinci-003 | 52.5 |
23
+
24
+ ## License
25
+ The code in this repo (what little there is of it) is Apache-2 licensed. The model weights have a `CC BY-SA 4.0` license. The TL;DR is that you can use and modify the model for any purpose – including commercial use. However, if you modify the weights (for example, by fine-tuning), you must open-source your modified weights under the same license terms.
26
+
27
+ ## Training
28
+ SQLCoder was trained on more than 20,000 human-curated questions. These questions were based on 10 different schemas. None of the schemas in the training data were included in our evaluation framework.
29
+
30
+ You can read more about our [training approach](https://defog.ai/blog/open-sourcing-sqlcoder2-7b/) and [evaluation framework](https://defog.ai/blog/open-sourcing-sqleval/).
31
+
32
+ ## Results by question category
33
+ We classified each generated question into one of 5 categories. The table displays the percentage of questions answered correctly by each model, broken down by category.
34
+ | query_category | gpt-4 | sqlcoder2-15b | sqlcoder-7b | gpt-3.5 | claude-2 | claude-instant | gpt-3 |
35
+ |:-----------------|--------:|----------------:|--------------:|----------:|-----------:|-----------------:|--------:|
36
+ | date | 72 | 76 | 64 | 68 | 52 | 48 | 32 |
37
+ | group_by | 91.4 | 80 | 82.9 | 77.1 | 71.4 | 71.4 | 71.4 |
38
+ | order_by | 82.9 | 77.1 | 74.3 | 68.6 | 74.3 | 74.3 | 68.6 |
39
+ | ratio | 80 | 60 | 54.3 | 37.1 | 57.1 | 45.7 | 25.7 |
40
+ | join | 82.9 | 77.1 | 74.3 | 71.4 | 65.7 | 62.9 | 57.1 |
41
+ | where | 80 | 77.1 | 74.3 | 74.3 | 62.9 | 60 | 54.3 |
42
+
43
+ ## Using SQLCoder
44
+ You can use SQLCoder via the `transformers` library by downloading our model weights from the Hugging Face repo. We have added sample code for [inference](./inference.py) on a [sample database schema](./metadata.sql).
45
+ ```bash
46
+ python inference.py -q "Question about the sample database goes here"
47
+
48
+ # Sample question:
49
+ # Do we get more revenue from customers in New York compared to customers in San Francisco? Give me the total revenue for each city, and the difference between the two.
50
+ ```
51
+
52
+ You can also use a demo on our website [here](https://defog.ai/sqlcoder-demo), or run SQLCoder in Colab [here](https://colab.research.google.com/drive/13BIKsqHnPOBcQ-ba2p77L5saiepTIwu0#scrollTo=ZpbVgVHMkJvC)
53
+
54
+ ## Hardware Requirements
55
+ SQLCoder has been tested on an A100 40GB GPU with `bfloat16` weights. You can also load an 8-bit and 4-bit quantized version of the model on consumer GPUs with 20GB or more of memory – like RTX 4090, RTX 3090, and Apple M2 Pro, M2 Max, or M2 Ultra Chips with 20GB or more of memory.
56
+
57
+ ## Todo
58
+
59
+ - [x] Open-source the v1 model weights
60
+ - [x] Train the model on more data, with higher data variance
61
+ - [ ] Tune the model further with Reward Modelling and RLHF
62
+ - [ ] Pretrain a model from scratch that specializes in SQL analysis