nreimers commited on
Commit
c167544
1 Parent(s): 94e0887

Add new SentenceTransformer model.

Browse files
1_Pooling/config.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "word_embedding_dimension": 384,
3
+ "pooling_mode_cls_token": false,
4
+ "pooling_mode_mean_tokens": true,
5
+ "pooling_mode_max_tokens": false,
6
+ "pooling_mode_mean_sqrt_len_tokens": false
7
+ }
README.md CHANGED
@@ -1,24 +1,42 @@
1
- # Sentence Embedding Model for MS MARCO Passage Retrieval
 
 
 
 
 
 
 
2
 
 
3
 
4
- This a `distilroberta-base` model from the [sentence-transformers](https://github.com/UKPLab/sentence-transformers)-repository. It was trained on the [MS MARCO Passage Retrieval dataset](https://github.com/microsoft/MSMARCO-Passage-Ranking): Given a search query, it finds the relevant passages.
5
 
6
- You can use this model for semantic search. Details can be found on: [SBERT.net - Semantic Search](https://www.sbert.net/examples/applications/semantic-search/README.html).
7
 
8
- This model was optimized to be used with **cosine-similarity** as similarity function between queries and documents.
9
 
 
 
 
 
 
 
 
10
 
11
- ## Training
 
 
 
 
12
 
13
- Details about the training of the models can be found here: [SBERT.net - MS MARCO](https://www.sbert.net/examples/training/ms_marco/README.html)
 
 
 
14
 
15
- ## Performance
16
 
17
- For performance details, see: [SBERT.net - Pre-Trained Models - MS MARCO](https://www.sbert.net/docs/pretrained-models/msmarco-v3.html)
18
 
19
- ## Usage (HuggingFace Models Repository)
 
20
 
21
- You can use the model directly from the model repository to compute sentence embeddings:
22
  ```python
23
  from transformers import AutoTokenizer, AutoModel
24
  import torch
@@ -28,67 +46,54 @@ import torch
28
  def mean_pooling(model_output, attention_mask):
29
  token_embeddings = model_output[0] #First element of model_output contains all token embeddings
30
  input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
31
- sum_embeddings = torch.sum(token_embeddings * input_mask_expanded, 1)
32
- sum_mask = torch.clamp(input_mask_expanded.sum(1), min=1e-9)
33
- return sum_embeddings / sum_mask
34
-
35
 
36
- # Queries we want embeddings for
37
- queries = ['What is the capital of France?', 'How many people live in New York City?']
38
 
39
- # Passages that provide answers
40
- passages = ['Paris is the capital of France', 'New York City is the most populous city in the United States, with an estimated 8,336,817 people living in the city, according to U.S. Census estimates dating July 1, 2019']
41
 
42
- #Load AutoModel from huggingface model repository
43
- tokenizer = AutoTokenizer.from_pretrained("model_name")
44
- model = AutoModel.from_pretrained("model_name")
45
 
46
- def compute_embeddings(sentences):
47
- #Tokenize sentences
48
- encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
49
 
50
- #Compute query embeddings
51
- with torch.no_grad():
52
- model_output = model(**encoded_input)
53
 
54
- #Perform pooling. In this case, mean pooling
55
- return mean_pooling(model_output, encoded_input['attention_mask'])
56
 
57
- query_embeddings = compute_embeddings(queries)
58
- passage_embeddings = compute_embeddings(passages)
59
  ```
60
 
61
- ## Usage (Sentence-Transformers)
62
- Using this model becomes more convenient when you have [sentence-transformers](https://github.com/UKPLab/sentence-transformers) installed:
63
- ```
64
- pip install -U sentence-transformers
65
- ```
66
 
67
- Then you can use the model like this:
68
- ```python
69
- from sentence_transformers import SentenceTransformer
70
- model = SentenceTransformer('model_name')
71
 
72
- # Queries we want embeddings for
73
- queries = ['What is the capital of France?', 'How many people live in New York City?']
74
 
75
- # Passages that provide answers
76
- passages = ['Paris is the capital of France', 'New York City is the most populous city in the United States, with an estimated 8,336,817 people living in the city, according to U.S. Census estimates dating July 1, 2019']
77
 
78
- query_embeddings = model.encode(queries)
79
- passage_embeddings = model.encode(passages)
80
- ```
81
 
82
- ## Changes in v3
83
- The models from v2 have been used for find for all training queries similar passages. An [MS MARCO Cross-Encoder](ce-msmarco.md) based on the electra-base-model has been then used to classify if these retrieved passages answer the question.
84
 
85
- If they received a low score by the cross-encoder, we saved them as hard negatives: They got a high score from the bi-encoder, but a low-score from the (better) cross-encoder.
86
 
87
- We then trained the v2 models with these new hard negatives.
 
 
 
 
 
 
88
 
89
  ## Citing & Authors
 
 
 
90
  If you find this model helpful, feel free to cite our publication [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://arxiv.org/abs/1908.10084):
91
- ```
92
  @inproceedings{reimers-2019-sentence-bert,
93
  title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
94
  author = "Reimers, Nils and Gurevych, Iryna",
 
1
+ ---
2
+ pipeline_tag: sentence-similarity
3
+ tags:
4
+ - sentence-transformers
5
+ - feature-extraction
6
+ - sentence-similarity
7
+ - transformers
8
+ ---
9
 
10
+ # sentence-transformers/msmarco-MiniLM-L-12-v3
11
 
12
+ This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 384 dimensional dense vector space and can be used for tasks like clustering or semantic search.
13
 
 
14
 
 
15
 
16
+ ## Usage (Sentence-Transformers)
17
+
18
+ Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
19
+
20
+ ```
21
+ pip install -U sentence-transformers
22
+ ```
23
 
24
+ Then you can use the model like this:
25
+
26
+ ```python
27
+ from sentence_transformers import SentenceTransformer
28
+ sentences = ["This is an example sentence", "Each sentence is converted"]
29
 
30
+ model = SentenceTransformer('sentence-transformers/msmarco-MiniLM-L-12-v3')
31
+ embeddings = model.encode(sentences)
32
+ print(embeddings)
33
+ ```
34
 
 
35
 
 
36
 
37
+ ## Usage (HuggingFace Transformers)
38
+ Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
39
 
 
40
  ```python
41
  from transformers import AutoTokenizer, AutoModel
42
  import torch
 
46
  def mean_pooling(model_output, attention_mask):
47
  token_embeddings = model_output[0] #First element of model_output contains all token embeddings
48
  input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
49
+ return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
 
 
 
50
 
 
 
51
 
52
+ # Sentences we want sentence embeddings for
53
+ sentences = ['This is an example sentence', 'Each sentence is converted']
54
 
55
+ # Load model from HuggingFace Hub
56
+ tokenizer = AutoTokenizer.from_pretrained('sentence-transformers/msmarco-MiniLM-L-12-v3')
57
+ model = AutoModel.from_pretrained('sentence-transformers/msmarco-MiniLM-L-12-v3')
58
 
59
+ # Tokenize sentences
60
+ encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
 
61
 
62
+ # Compute token embeddings
63
+ with torch.no_grad():
64
+ model_output = model(**encoded_input)
65
 
66
+ # Perform pooling. In this case, max pooling.
67
+ sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
68
 
69
+ print("Sentence embeddings:")
70
+ print(sentence_embeddings)
71
  ```
72
 
 
 
 
 
 
73
 
 
 
 
 
74
 
75
+ ## Evaluation Results
 
76
 
 
 
77
 
 
 
 
78
 
79
+ For an automated evaluation of this model, see the *Sentence Embeddings Benchmark*: [https://seb.sbert.net](https://seb.sbert.net?model_name=sentence-transformers/msmarco-MiniLM-L-12-v3)
80
+
81
 
 
82
 
83
+ ## Full Model Architecture
84
+ ```
85
+ SentenceTransformer(
86
+ (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel
87
+ (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
88
+ )
89
+ ```
90
 
91
  ## Citing & Authors
92
+
93
+ This model was trained by [sentence-transformers](https://www.sbert.net/).
94
+
95
  If you find this model helpful, feel free to cite our publication [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://arxiv.org/abs/1908.10084):
96
+ ```bibtex
97
  @inproceedings{reimers-2019-sentence-bert,
98
  title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
99
  author = "Reimers, Nils and Gurevych, Iryna",
config.json CHANGED
@@ -1,5 +1,5 @@
1
  {
2
- "_name_or_path": "../output/microsoft_MiniLM-L12-H384-uncased-mined_hard_neg-mean-pooling-no_identifier-epoch10-batchsize70-2021-04-05_07-37-55/0_Transformer",
3
  "architectures": [
4
  "BertModel"
5
  ],
@@ -17,7 +17,7 @@
17
  "num_hidden_layers": 12,
18
  "pad_token_id": 0,
19
  "position_embedding_type": "absolute",
20
- "transformers_version": "4.4.2",
21
  "type_vocab_size": 2,
22
  "use_cache": true,
23
  "vocab_size": 30522
 
1
  {
2
+ "_name_or_path": "old_models/msmarco-MiniLM-L-12-v3/0_Transformer",
3
  "architectures": [
4
  "BertModel"
5
  ],
 
17
  "num_hidden_layers": 12,
18
  "pad_token_id": 0,
19
  "position_embedding_type": "absolute",
20
+ "transformers_version": "4.7.0",
21
  "type_vocab_size": 2,
22
  "use_cache": true,
23
  "vocab_size": 30522
config_sentence_transformers.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "__version__": {
3
+ "sentence_transformers": "2.0.0",
4
+ "transformers": "4.7.0",
5
+ "pytorch": "1.9.0+cu102"
6
+ }
7
+ }
modules.json ADDED
@@ -0,0 +1,14 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "idx": 0,
4
+ "name": "0",
5
+ "path": "",
6
+ "type": "sentence_transformers.models.Transformer"
7
+ },
8
+ {
9
+ "idx": 1,
10
+ "name": "1",
11
+ "path": "1_Pooling",
12
+ "type": "sentence_transformers.models.Pooling"
13
+ }
14
+ ]
pytorch_model.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:57555aab9e3b37aa1e185d6e4534fe279867ec793aebf9e5176294dcb2519114
3
- size 133526519
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b17442e1b4a5db687084be01f47d35f683f35ffadbfe26ca42622fec51d013a8
3
+ size 133518577
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json CHANGED
@@ -1 +1 @@
1
- {"do_lower_case": true, "unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]", "tokenize_chinese_chars": true, "strip_accents": null, "model_max_length": 512, "name_or_path": "microsoft/MiniLM-L12-H384-uncased", "do_basic_tokenize": true, "never_split": null}
 
1
+ {"do_lower_case": true, "unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]", "tokenize_chinese_chars": true, "strip_accents": null, "model_max_length": 512, "name_or_path": "old_models/msmarco-MiniLM-L-12-v3/0_Transformer", "do_basic_tokenize": true, "never_split": null, "special_tokens_map_file": "old_models/msmarco-MiniLM-L-12-v3/0_Transformer/special_tokens_map.json"}