Trelis

/

Llama-2-7b-chat-hf-function-calling-v3

like 35

vLLM can expose an openai like api and can I just ues the openai to make function calling? Right now I have a try like this, and it failed.

import json

# Example dummy function hard coded to return the same weather
# In production, this could be your backend API or an external API
def get_current_weather(location, unit="fahrenheit"):
    """Get the current weather in a given location"""
    weather_info = {
        "location": location,
        "temperature": "72",
        "unit": unit,
        "forecast": ["sunny", "windy"],
    }
    return json.dumps(weather_info)

# define a function
functions = [
    {
        "name": "get_current_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {
                    "type": "string",
                    "description": "The city and state, e.g. San Francisco, CA",
                },
                "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
            },
            "required": ["location"],
        },
    }
]

client = OpenAI(api_key='abc', base_url='<customized-enpoint>')

response = client.chat.completions.create(messages=messages, functions=functions, model='Llama-2-7b-chat-hf-function-calling-v3')

The result looks like this:

The weather in Boston can vary depending on the time of year. Boston has a humid continental climate, with cold winters and hot, humid summers. Here are some general weather patterns in Boston:

Winter (December to February):

* Temperatures can range from 20°F (-7°C) to 40°F (4°C)
* Snowfall is common, with an average of 43 inches (109 cm) per year
* The coldest months are January and February, with average temperatures around 28°F (-2°C)

Spring (March to May):

* Temperatures can range from 30°F (-1°C) to 60°F (16°C)
* Spring is a transition season, with temperatures gradually warming up
* April is usually the rainiest month, with an average of 4 inches (10 cm) of rain

Summer (June to August):

* Temperatures can range from 60°F (16°C) to 80°F (27°C)
* Summer is the warmest season, with an average temperature of 70°F (21°C) in July and August
* Humidity is usually high during the summer months

Fall (September to November):

* Temperatures can range from 50°F (10°C) to 60°F (16°C)
* Fall is a mild season, with temperatures gradually cooling down
* October is usually the driest month, with an average of 3 inches (7 cm) of rain

It's worth noting that weather patterns can vary from year to year, and it's not uncommon for Boston to experience extreme weather events such as heatwaves, cold snaps, or heavy snowfall.