How to fine tune ChatGPT

Fine-tuning Model

Are you looking for a way to make ChatGPT better for your needs? With some fine-tuning, you can tailor ChatGPT to suit your specific requirements. In this blog, we will explore the fascinating world of customizing ChatGPT, enabling you to unleash its full potential.

Whether you're a developer, researcher, or AI enthusiast, this blog will provide valuable insights and practical guidance on how to enhance ChatGPT to meet your unique demands.

What is Fine-tuning chatGPT?

Fine-tuning ChatGPT is a process of customizing and adapting the pre-trained language model to perform specific tasks or cater to specific requirements. The initial version of ChatGPT is trained on a large corpus of diverse text data to generate creative and coherent responses. However, fine-tuning allows you to refine its behavior and make it more suitable for your specific application or use case.

Fine-tuning improves on few-shot learning by training on many more examples than can fit in the prompt, letting you achieve better results on a wide number of tasks. Once a model has been fine-tuned, you won't need to provide examples in the prompt anymore. This saves costs and enables lower-latency requests.

At a high level, fine-tuning involves the following steps:

• Prepare and upload training data
• Train a new fine-tuned model
• Use your fine-tuned model

What chatgpt models can be fine-tuned?

• davinci
• curie
• babbage
• ada
• text-davinci-003

How to fine tune chatgpt model guide

• Installation of OpenAI CLI

  Open you CLI and enter thsi command :pip install --upgrade openaiOpenAI CLI requires Python

  Now next Set your OPENAI_API_KEY environment variable by adding the following line into your shell initialization script (e.g. .bashrc, zshrc, etc.) or running it in the command line before the fine-tuning command: export OPENAI_API_KEY=""

• Prepare training data

  The way you teach GPT-3 what you want it to say is by training data. Your input must be a JSONL document with each line containing a pair of prompts and answers that correspond to a training example. To quickly convert your data into this file format, utilise our CLI data preparation tool.

  {"prompt": "", "completion": ""} {"prompt": "", "completion": ""} {"prompt": "", "completion": ""} ...

  When creating your prompts and completions for fine-tuning, you should design them differently from when creating them for our base models (Da Vinci, Curie, Babbage, and Ada). In particular, while base models frequently use multiple examples in their prompts ("few-shot learning"), fine-tuning typically only requires a single input example and its corresponding output, negating the need for lengthy instructions or multiple examples in a single prompt.

  Please refer to our preparing the dataset best practises for more specific advice on how to put up training data for different jobs.

  It is best to have as many training examples as possible. We advise having a minimum of a few hundred examples. We have discovered that the model quality increases linearly with each doubling of the dataset size.

• CLI data preparation tool

  Now prepare the dataset you want to teach and train gpt model in form of json. \ {"prompt": "", "completion": ""} {"prompt": "", "completion": ""} {"prompt": "", "completion": ""} ...

  Use this tool developed to validates, suggestions and reformats your data: openai tools fine_tunes.prepare_data -f
• How to create a fine-tuned model

  Start your fine-tuning job using the OpenAI CLI:

  openai api fine_tunes.create -t -m

  Running the above command does several things:

  • Uploads the file using the files API (or uses an already-uploaded file)
  • Creates a fine-tune job
  • Streams events until the job is done (this often takes minutes, but can take hours if there are many jobs in the queue or your dataset is large)

• How to use a fine-tuned model

  When a job has succeeded, the fine_tuned_model field will be populated with the name of the model. Now, you can use the Playground to send calls to our Completions API and specify this model as a parameter.

  It can take a while for your model to be prepared to receive requests after your job has finished. It is possible that your model is still being loaded if completion requests to it time out. Try again in a few minutes if this occurs.

  You can start making requests by passing the model name as the model parameter of a completion request:
  1. OpenAI CLI:

     openai api completions.create -m -p

  2. cURL:

     curl https://api.openai.com/v1/completions \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -H "Content-Type: application/json" \ -d '{"prompt": YOUR_PROMPT, "model": FINE_TUNED_MODEL}'

  3. Python: import openai openai.Completion.create( model=FINE_TUNED_MODEL, prompt=YOUR_PROMPT)
  4. Node.js:: const response = await openai.createCompletion({ model: FINE_TUNED_MODEL prompt: YOUR_PROMPT, });

  On these requests to fine-tune models, you may continue to use all the other Completions parameters, such as temperature, frequency_penalty, presence_penalty, etc.

• How to Delete a fine-tuned model You must be recognised as a "owner" inside your organisation in order to delete a tuned model.
  OpenAI CLI: openai api models.delete -i cURL: curl -X "DELETE" https://api.openai.com/v1/models/ \ -H "Authorization: Bearer $OPENAI_API_KEY" Python: import openai openai.Model.delete(FINE_TUNED_MODEL)
• Preparing your dataset

  Fine-tuning is a powerful technique to create a new model that's specific to your use case. Before fine-tuning your model, we strongly recommend reading these best practices and specific guidelines for your use case below.

  Data formatting

  To fine-tune a model, you'll need a set of training examples that each consist of a single input ("prompt") and its associated output ("completion"). This is notably different from using our base models, where you might input detailed instructions or multiple examples in a single prompt.

  • Each prompt should end with a fixed separator to inform the model when the prompt ends and the completion begins. A simple separator which generally works well is \n\n###\n\n. The separator should not appear elsewhere in any prompt.
  • Each completion should start with a whitespace due to our tokenization, which tokenizes most words with a preceding whitespace.
  • Each completion should end with a fixed stop sequence to inform the model when the completion ends. A stop sequence could be \n, ###, or any other token that does not appear in any completion.
  • For inference, you should format your prompts in the same way as you did when creating the training dataset, including the same separator. Also specify the same stop sequence to properly truncate the completion.

  fine tuning chatgpt example

  Here is an example of how you could fine-tune ChatGPT to generate marketing copy:
  • Select a pre-trained ChatGPT model that is relevant to marketing.
  • Collect a dataset of labeled examples of marketing copy.
  • Fine-tune the model on your dataset using a supervised learning approach.
  • Deploy the model to production and use it to generate marketing copy.
  
  

  Fine-tuning ChatGPT involves training the base language model with additional data specific to the desired application or domain, allowing it to generate more contextually relevant and accurate responses.

  Fine Tuning model General best practices

  Fine-tuning performs better with more high-quality examples. To fine-tune a model that performs better than using a high-quality prompt with our base models,
  you should provide at least a few hundred high-quality examples, ideally vetted by human experts. From there, performance tends to linearly increase with every doubling of the number of examples. Increasing the number of examples is usually the best and most reliable way of improving performance.
  Classifiers are the easiest models to get started with. For classification problems we suggest using ada, which generally tends to perform only very slightly worse than more capable models once fine-tuned, whilst being significantly faster and cheaper.
  If you are fine-tuning on a pre-existing dataset rather than writing prompts from scratch, be sure to manually review your data for offensive or inaccurate content if possible, or review as many random samples of the dataset as possible if it is large.

Fine tuning model Specific guidelines

Fine-tuning can solve a variety of problems, and the optimal way to use it may depend on your specific use case. Below, we've listed the most common use cases for fine-tuning and corresponding guidelines.

• Classification

  In classification problems, each input in the prompt should be classified into one of the predefined classes. For this type of problem, we recommend:

  Use a separator at the end of the prompt, e.g. \n\n###\n\n. Remember to also append this separator when you eventually make requests to your model.
  Choose classes that map to a single token. At inference time, specify max_tokens=1 since you only need the first token for classification.
  Ensure that the prompt + completion doesn't exceed 2048 tokens, including the separator
  Aim for at least ~100 examples per class
  To get class log probabilities you can specify logprobs=5 (for 5 classes) when using your model
  Ensure that the dataset used for finetuning is very similar in structure and type of task as what the model will be used for

  Is the model making untrue statements?

  Case study: Is the model making untrue statements?

  A good example is when you want to maintain accurate ads on your website, consider fine-tuning a classifier that filters out incorrect product and company mentions like this,

  The dataset might look something like the following:

  {"prompt":"Company: BHFF insurance\nProduct: allround insurance\nAd:One stop shop for all your insurance needs!\nSupported:", "completion":" yes"} {"prompt":"Company: Loft conversion specialists\nProduct: -\nAd:Straight teeth in weeks!\nSupported:", "completion":" no"}

  Now we can query our model by making a Completion request.

  curl https://api.openai.com/v1/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d '{ "prompt": "Company: Reliable accountants Ltd\nProduct: Personal Tax help\nAd:Best advice in town!\nSupported:", "max_tokens": 1, "model": "YOUR_FINE_TUNED_MODEL_NAME" }' Which will return either yes or no.
• Model Case study: Sentiment analysis

  Let's say you'd like to get a degree to which a particular tweet is positive or negative. The dataset might look something like the following:

  {"prompt":"Overjoyed with the new iPhone! ->", "completion":" positive"} {"prompt":"@lakers disappoint for a third straight night https://t.co/38EFe43 ->", "completion":" negative"}

  Now we can query our model by making a Completion request.

  curl https://api.openai.com/v1/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d '{ "prompt": "https://t.co/f93xEd2 Excited to share my latest blog post! ->", "max_tokens": 1, "model": "YOUR_FINE_TUNED_MODEL_NAME" }'

  Will return :

  { "id": "cmpl-COMPLETION_ID", "object": "text_completion", "created": 1589498378, "model": "YOUR_FINE_TUNED_MODEL_NAME", "choices": [ { "logprobs": { "text_offset": [ 19 ], "token_logprobs": [ -0.03597255 ], "tokens": [ " positive" ], "top_logprobs": [ { " negative": -4.9785037, " positive": -0.03597255 } ] }, "text": " positive", "index": 0, "finish_reason": "length" } ] }

Fine tuning conclusion

Fine-tuning model the ChatGPT model has proven to be effective in enhancing its conversational abilities. The model can produce more accurate and contextually relevant replies by being trained on particular datasets and tailored for particular activities. But it's crucial to create a balance between perfecting and averting prejudices or incorrect data. To guarantee that the model's performance complies with the specified goals, regular review and monitoring are essential. Overall, with some tweaking, ChatGPT is able to be an adaptable and flexible language model that may provide useful and enjoyable conversational experiences.