Update README.md

README.md

@@ -10,74 +10,114 @@ pinned: true
 license: apache-2.0
 short_description: Pearl-7B, an xtraordinary Space
 ---
-
-# MANATEE(lm) : Market Analysis based on language model architectures
-[![Python](https://img.shields.io/pypi/pyversions/tensorflow.svg)](https://badge.fury.io/py/tensorflow) [![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) ![Maintainer](https://img.shields.io/badge/maintainer-@louisbrulenaudet-blue)
-
-This project focuses on employing LLM to analyze time series data for forecasting purposes, based on the "Chronos: Learning the Language of Time Series" paper from the Amazon Web Services and Amazon Supply Chain Optimization Technologies. The MANATEE project is designed to fetch, compute, and plot historical data for financial securities, leveraging APIs from Alpaca and the power of Polars and Plotly for data manipulation and visualization. With features like calculating the rolling mean and Relative Strength Index (RSI), this tool also aids in analyzing the past performance of stocks and crypto assets.
-
-![Plot](https://github.com/louisbrulenaudet/manatee/blob/main/scatter.png?raw=true)
-
-From source :
-> In this work, we take a step back and ask: what are the fundamental differences between a language model that predicts the next token, and a time series forecasting model that predicts the next values? Despite the apparent distinction — tokens from a finite dictionary versus values from an unbounded, usually continuous domain — both endeavors fundamentally aim to model the sequential structure of the data to predict future patterns. Shouldn't good language models “just work” on time series? This naive question prompts us to challenge the necessity of time-series-specific modifications, and answering it led us to develop Chronos, a language modeling framework minimally adapted for time series forecasting. Chronos tokenizes time series into discrete bins through simple scaling and quantization of real values. In this way, we can train off-the-shelf language models on this “language of time series,” with no changes to the model architecture. Remarkably, this straightforward approach proves to be effective and efficient, underscoring the potential for language model architectures to address a broad range of time series problems with minimal modifications.
-[...]
-
-## Dependencies
-### Libraries Used:
-
-1. **`json`**: A built-in Python library for parsing JSON data. No need for installation.
-
-2. **`datetime` & `time`**: Built-in Python libraries for handling date and time. Used here for defining time frames for data fetching. No installation required.
-
-3. **`plotly`** (as `px`): Provides an easy-to-use interface to Plotly, which is used for creating interactive plots. Install via pip:
-   ```shell
-   pip3 install plotly
-   ```
-   
-4. **`polars`** (as `pl`): A fast DataFrames library ideal for financial time-series data. Install using pip:
-   ```shell
-   pip3 install polars
-   ```
-   
-5. **`alpaca-py`**: A Python library for Alpaca API. It provides access to historical stock/crypto data and trading operations. Install using pip:
-   ```shell
-   pip3 install alpaca-trade-api
-   ```
-
-### Installation Guide
-
-To install all the dependencies, you can use the following command:
-
-```shell
-pip3 install plotly polars alpaca-py transformers gradio spaces
+<center><img src='https://i.imgur.com/0xFTuAX.png' width='450px'></center>
+
+# Pearl-7B-0211-ties, an xtraordinary 7B model
+
+**03-22-2024 - To date, louisbrulenaudet/Pearl-34B-ties is the "Best 🤝 base merges and moerges model of around 30B" on the Open LLM Leaderboard.**
+
+Pearl-7B-0211-ties is a merge of the following models:
+* [louisbrulenaudet/Pearl-7B-slerp](https://huggingface.co/louisbrulenaudet/Pearl-7B-slerp)
+* [WizardLM/WizardMath-7B-V1.1](https://huggingface.co/WizardLM/WizardMath-7B-V1.1)
+* [cognitivecomputations/WestLake-7B-v2-laser](https://huggingface.co/cognitivecomputations/WestLake-7B-v2-laser)
+* [CultriX/NeuralTrix-7B-dpo](https://huggingface.co/CultriX/NeuralTrix-7B-dpo)
+
+## Evaluation
+
+The evaluation was performed using the HuggingFace Open LLM Leaderboard.
+
+| Model                                            | Average | ARC   | HellaSwag | MMLU  | TruthfulQA | Winogrande | GSM8K | #Params (B) |
+|--------------------------------------------------|---------|-------|-----------|-------|------------|------------|-------|--------------|
+| **louisbrulenaudet/Pearl-34B-ties**             | **75.48** | 70.99 | 84.83 | **76.63** | 70.32 | 82.64 | 67.48 | 34.39 |
+| **louisbrulenaudet/Pearl-7B-0211-ties**         | **75.11** | **71.42** | **88.86** | 63.91 | **71.46** | **84.37** | 70.66 | 7.24 |
+| NousResearch/Nous-Hermes-2-Mixtral-8x7B-DPO     | 73.35   | 71.08 | 87.29 | 72.17 | 54.83 | 83.11 | 71.65 | 46.7 |
+| argilla/notus-8x7b-experiment                   | 73.18   | 70.99 | 87.73 | 71.33 | 65.79 | 81.61 | 61.64 | 46.7 |
+| **louisbrulenaudet/Pearl-7B-slerp**             | 72.75   | 68.00 | 87.16 | 64.04 | 62.35 | 81.29 | **73.62** | 7.24 |
+| mistralai/Mixtral-8x7B-Instruct-v0.1            | 72.7    | 70.14 | 87.55 | 71.4  | 64.98 | 81.06 | 61.11 | 46.7 |
+| microsoft/Orca-2-13b                            | 61.98   | 60.92 | 79.85 | 60.3  | 56.42 | 76.56 | 37.83 | 13 |
+| microsoft/phi-2                                 | 61.33   | 61.09 | 75.11 | 58.11 | 44.47 | 74.35 | 54.81 | 2.78 |
+
+### Ties merging
+
+TIES-Merging is a method designed to facilitate the efficient merging of multiple task-specific models into a consolidated multitask model. It addresses two primary challenges encountered in the process of model merging with a focus on maintaining objectivity.
+
+One key challenge tackled by TIES-Merging involves addressing redundancy in model parameters. This is achieved by identifying and eliminating redundant parameters within task-specific models, emphasizing the changes made during fine-tuning and selectively retaining the top-k% most significant changes while discarding the rest.
+
+Another challenge pertains to conflicts arising from disagreements between parameter signs across different models. TIES-Merging resolves these conflicts by creating a unified sign vector representing the most dominant direction of change across all models.
+
+The TIES-Merging process consists of three steps:
+
+- Trim: Reduces redundancy in task-specific models by retaining a fraction of the most significant parameters (density parameter) and resetting the remaining parameters to zero.
+- Elect Sign: Resolves sign conflicts across different models by creating a unified sign vector based on the most dominant direction (positive or negative) in terms of cumulative magnitude.
+- Disjoint Merge: Averages parameter values aligned with the unified sign vector, excluding zero values.
+
+## Configuration
+
+```yaml
+models:
+  - model: OpenPipe/mistral-ft-optimized-1227
+  - model: louisbrulenaudet/Pearl-7B-slerp
+    parameters:
+      density: 0.6
+      weight: 0.3
+  - model: WizardLM/WizardMath-7B-V1.1
+    parameters:
+      density: 0.55
+      weight: 0.2
+  - model: cognitivecomputations/WestLake-7B-v2-laser
+    parameters:
+      density: 0.55
+      weight: 0.25
+  - model: CultriX/NeuralTrix-7B-dpo
+    parameters:
+     density: 0.6
+     weight: 0.25
+merge_method: ties
+base_model: OpenPipe/mistral-ft-optimized-1227
+parameters:
+  normalize: true
+  int8_mask: true
+dtype: float16
 ```
 
-Note: Ensure you have Python installed on your system before proceeding with the installation of these libraries.
+## Usage
 
-## Best Practices
-- **API Keys Management**: For security reasons, avoid hardcoding your API keys into the script. Consider using environment variables or a secure vault service.
+```python
+!pip install -qU transformers accelerate
 
-- **Data Privacy**: When handling financial data, it's crucial to comply with data protection regulations (such as GDPR, CCPA). Ensure you have the right to use and share the data fetched through this tool.
+from transformers import AutoTokenizer
+import transformers
+import torch
 
-- **Error Handling**: The script includes basic error handling, but for production use, consider implementing more comprehensive try-except blocks to handle network errors, API limit exceptions, and data inconsistencies.
+model = "louisbrulenaudet/Pearl-7B-0211-ties"
+messages = [{"role": "user", "content": "What is a large language model?"}]
 
-- **Plotting Considerations**: This tool uses Plotly for visualization, which is very versatile but can be resource-intensive for large datasets. For analyzing large datasets, consider creating plots with fewer data points or aggregating the data before plotting.
+tokenizer = AutoTokenizer.from_pretrained(model)
+prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+pipeline = transformers.pipeline(
+    "text-generation",
+    model=model,
+    torch_dtype=torch.float16,
+    device_map="auto",
+)
 
-- **Resource Management**: When dealing with large datasets or numerous API requests, monitor your system's and the API's usage to avoid overloading.
+outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
+print(outputs[0]["generated_text"])
+```
 
-- **Version Control**: Regularly update your dependencies. Financial APIs and data handling libraries evolve, and keeping them up to date can improve security, efficiency, and accessibility of new features.
+## Citing & Authors
 
-## Citing this project
 If you use this code in your research, please use the following BibTeX entry.
 
 ```BibTeX
-@misc{louisbrulenaudet2023,
-	author = {Louis Brulé Naudet},
-	title = {MANATEE(lm) : Market Analysis based on language model architectures},
-	howpublished = {\url{https://huggingface.co/spaces/louisbrulenaudet/manatee}},
-	year = {2024}
+@misc{louisbrulenaudet2024,
+  author =       {Louis Brulé Naudet},
+  title =        {Pearl-7B-0211-ties, an xtraordinary 7B model},
+  year =         {2024}
+  howpublished = {\url{https://huggingface.co/louisbrulenaudet/Pearl-7B-0211-ties}},
 }
-
 ```
+
 ## Feedback
-If you have any feedback, please reach out at [louisbrulenaudet@icloud.com](mailto:louisbrulenaudet@icloud.com).
+
+If you have any feedback, please reach out at [louisbrulenaudet@icloud.com](mailto:louisbrulenaudet@icloud.com).