examples/BaiChuan13B_NER——transformers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4e7bef94-e459-4140-b533-cd6c188722b0",
   "metadata": {},
   "source": [
    "# Baichuan-13B 命名实体识别微调"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5c26727-c524-46af-a0cf-389a389a5ab5",
   "metadata": {},
   "source": [
    "😋😋公众号算法美食屋后台回复关键词：**torchkeras**，获取本文notebook源代码和数据集下载链接。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "007b2845-a8a5-44e1-bc5e-404fb2209af2",
   "metadata": {},
   "source": [
    "传统上，一般把NLP的研究领域大致分为自然语言理解（NLU）和自然语言生成（NLG）两种。\n",
    "\n",
    "NLU侧重于如何理解文本，包括文本分类、命名实体识别、指代消歧、句法分析、机器阅读理解等；\n",
    "\n",
    "NLG则侧重于理解文本后如何生成自然文本，包括自动摘要、机器翻译、问答系统、对话机器人等。\n",
    "\n",
    "但是以ChatGPT为代表的大模型出来后，这些传统的NLP的细分研究领域基本可以说都失去了独立研究的价值。\n",
    "\n",
    "为什么呢？因为大模型可以用统一的范式通通将它们搞定，并且效果非常出众。\n",
    "\n",
    "\n",
    "在之前的例子中，我们演示了使用QLoRA算法来对BaiChuan-13B实施微调以处理最简单的文本分类任务。\n",
    "\n",
    "在外卖评论数据集上，微调后测试集acc由0.8925提升到0.9015约提升了1个百分点。\n",
    "\n",
    "在本例中，我们使用几乎相同的流程和方法来微调BaiChuan-13B以更好地处理命名实体识别任务。\n",
    "\n",
    "实验结果显示，微调前，我们的f1-score仅仅是0.4313，微调后，取得了明显的提升，达到了0.8768。\n",
    "\n",
    "注：跑完本流程需要至少32G的CPU，需要约2个小时的训练时间。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ec8257f-9f37-4a33-b3ef-feedd23b71fa",
   "metadata": {},
   "source": [
    "在我们正式开始之前，请允许我用简短的话给没有NLP基础知识的小伙伴讲解一下什么是命名实体识别。\n",
    "\n",
    "命名实体识别NER任务是NLP的一个常见基础任务，\n",
    "\n",
    "它是Named Entity Recognization的简称。\n",
    "\n",
    "简单地说，就是识别一个句子中的各种 名称实体，诸如：人名，地名，机构 等。\n",
    "\n",
    "例如对于下面这句话：\n",
    "\n",
    "```\n",
    "小明对小红说:\"你听说过安利吗？\"\n",
    "```\n",
    "\n",
    "其命名实体可以抽取表示如下：\n",
    "\n",
    "```\n",
    "{\"人名\": [\"小明\",\"小红\"], \"组织\": [\"安利\"]}\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ee10aa6-bd22-4014-8fa4-7f5a49c884b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "#安装环境\n",
    "\n",
    "#baichuan-13b-chat\n",
    "#!pip install 'transformers==4.30.2'\n",
    "#!pip install  -U transformers_stream_generator\n",
    "\n",
    "\n",
    "#finetune\n",
    "#!pip install datasets\n",
    "#!pip install git+https://github.com/huggingface/accelerate\n",
    "#!pip install  git+https://github.com/huggingface/peft\n",
    "#!pip install  git+https://github.com/lyhue1991/torchkeras \n",
    "#!pip install 'bitsandbytes==0.39.1' #4bit量化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7480cac9-517d-40ee-8cc2-ff579daa7c94",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1cab03f4-4824-4ccd-817a-92fa4086b844",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "eae3a0b3-bf03-46bd-bace-330665645f9a",
   "metadata": {},
   "source": [
    "## 〇，预训练模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e633e3b-e4cc-48d4-8792-ea269c7396f6",
   "metadata": {},
   "source": [
    "我们需要从 https://huggingface.co/baichuan-inc/Baichuan-13B-Chat 下载baichuan-13b-chat的模型。\n",
    "\n",
    "国内可能速度会比较慢，总共有25个G左右，网速不太好的话，大概可能需要两到三个小时。\n",
    "\n",
    "如果网络不稳定，也可以手动从这个页面一个一个下载全部文件然后放置到 一个文件夹中例如 'baichuan-13b' 以便读取。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "9c0f2e17-7ed1-42e8-81f8-63a7bbeab31e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e200470d-5cec-44fd-a733-b26b7ff43a63",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2023-07-22 20:32:37,605] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)\n",
      "\n",
      "===================================BUG REPORT===================================\n",
      "Welcome to bitsandbytes. For bug reports, please run\n",
      "\n",
      "python -m bitsandbytes\n",
      "\n",
      " and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues\n",
      "================================================================================\n",
      "bin /usr/local/lib/python3.8/dist-packages/bitsandbytes/libbitsandbytes_cuda116.so\n",
      "CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...\n",
      "CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so\n",
      "CUDA SETUP: Highest compute capability among GPUs detected: 8.0\n",
      "CUDA SETUP: Detected CUDA version 116\n",
      "CUDA SETUP: Loading binary /usr/local/lib/python3.8/dist-packages/bitsandbytes/libbitsandbytes_cuda116.so...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "507938bf269b40dc86af92e285e10f82",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import torch\n",
    "from transformers import AutoTokenizer, AutoModelForCausalLM,AutoConfig, AutoModel, BitsAndBytesConfig\n",
    "from transformers.generation.utils import GenerationConfig\n",
    "import torch.nn as nn\n",
    "\n",
    "\n",
    "#使用QLoRA引入的 NF4量化数据类型以节约显存\n",
    "model_name_or_path ='../baichuan-13b' #远程 'baichuan-inc/Baichuan-13B-Chat'\n",
    "\n",
    "bnb_config=BitsAndBytesConfig(\n",
    "            load_in_4bit=True,\n",
    "            bnb_4bit_compute_dtype=torch.float16,\n",
    "            bnb_4bit_use_double_quant=True,\n",
    "            bnb_4bit_quant_type=\"nf4\",\n",
    "            llm_int8_threshold=6.0,\n",
    "            llm_int8_has_fp16_weight=False,\n",
    "        )\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(\n",
    "   model_name_or_path, trust_remote_code=True)\n",
    "\n",
    "model = AutoModelForCausalLM.from_pretrained(model_name_or_path,\n",
    "                quantization_config=bnb_config,\n",
    "                trust_remote_code=True) \n",
    "\n",
    "model.generation_config = GenerationConfig.from_pretrained(model_name_or_path)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "76722209-62a4-4e17-a672-9383791cd014",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "乔戈里峰。世界第二高峰———乔戈里峰海拔高度：8610米坐标：35.5°n,76.5°e乔戈里，蒙古语意为“高大雄伟”，它位于喀喇昆仑山脉的中部。\n"
     ]
    }
   ],
   "source": [
    "from IPython.display import clear_output \n",
    "messages = []\n",
    "messages.append({\"role\": \"user\",\n",
    "                 \"content\": \"世界上第二高的山峰是哪座?\"})\n",
    "response = model.chat(tokenizer,messages=messages,stream=True)\n",
    "for res in response:\n",
    "    print(res)\n",
    "    clear_output(wait=True)\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c754a8e9-c9d6-49c4-8a43-daafc8374129",
   "metadata": {},
   "source": [
    "下面我们设计一个7-shot-prompt方法，测试一下BaiChuan13b的实体抽取能力。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "81c1e15f-2217-4a76-8caa-945362f98317",
   "metadata": {},
   "outputs": [],
   "source": [
    "prefix = '''命名实体识别：抽取文本中的 人名，地点，组织 这三类命名实体，并按照json格式返回结果。\n",
    "\n",
    "下面是一些范例：\n",
    "\n",
    "小明对小红说:\"你听说过安利吗？\" -> {\"人名\": [\"小明\",\"小红\"], \"组织\": [\"安利\"]}\n",
    "现在，每年有几十万中国人到美国访问，几千名中国留学生到美国就学。 -> {\"地点\": [\"中国\", \"美国\"]}\n",
    "中国是联合国安理会常任理事国之一。 -> {\"地点\": [\"中国\"], \"组织\": [\"联合国\"]}\n",
    "\n",
    "请对下述文本进行实体抽取，返回json格式。\n",
    "\n",
    "'''\n",
    "\n",
    "def get_prompt(text):\n",
    "    return prefix+text+' -> '\n",
    "\n",
    "def get_message(prompt,response):\n",
    "    return [{\"role\": \"user\", \"content\": f'{prompt} -> '},\n",
    "            {\"role\": \"assistant\", \"content\": response}]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9974c379-3bf3-4089-b656-3d0861b1e76a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"地点\":[\"摩洛哥\"], \"组织\":[]}\n"
     ]
    }
   ],
   "source": [
    "messages  = [{\"role\": \"user\", \"content\": get_prompt(\"一些摩洛哥球迷已按捺不住，在看台上欢呼雀跃\")}]\n",
    "response = model.chat(tokenizer, messages)\n",
    "print(response)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "96c7d5d3-4ec0-47f4-8592-6cc77e7d54a8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'role': 'user',\n",
       "  'content': '命名实体识别：抽取文本中的 人名，地点，组织 这三类命名实体，并按照json格式返回结果。\\n\\n下面是一些范例：\\n\\n小明对小红说:\"你听说过安利吗？\" -> {\"人名\": [\"小明\",\"小红\"], \"组织\": [\"安利\"]}\\n现在，每年有几十万中国人到美国访问，几千名中国留学生到美国就学。 -> {\"地点\": [\"中国\", \"美国\"]}\\n中国是联合国安理会常任理事国之一。 -> {\"地点\": [\"中国\"], \"组织\": [\"联合国\"]}\\n\\n请对下述文本进行实体抽取，返回json格式。\\n\\n一些摩洛哥球迷已按捺不住，在看台上欢呼雀跃 -> '},\n",
       " {'role': 'assistant', 'content': \"{'地点': ['摩洛哥']}\"},\n",
       " {'role': 'user', 'content': '这次轮到北京国安队，不知会不会再步后尘？ -> '},\n",
       " {'role': 'assistant', 'content': \"{'组织': ['北京国安队']}\"},\n",
       " {'role': 'user', 'content': '革命党人孙中山在澳门成立同盟会分会 -> '},\n",
       " {'role': 'assistant',\n",
       "  'content': \"{'人名': ['孙中山'], '地名': ['澳门'], '组织': ['同盟会']}\"},\n",
       " {'role': 'user', 'content': '我曾在安徽芜湖市和上海浦东打工。 -> '},\n",
       " {'role': 'assistant', 'content': \"{'地点': ['安徽芜湖市', '上海浦东']}\"}]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "messages = messages+[{\"role\": \"assistant\", \"content\": \"{'地点': ['摩洛哥']}\"}]\n",
    "messages.extend(get_message(\"这次轮到北京国安队，不知会不会再步后尘？\",\"{'组织': ['北京国安队']}\"))\n",
    "messages.extend(get_message(\"革命党人孙中山在澳门成立同盟会分会\",\"{'人名': ['孙中山'], '地名': ['澳门'], '组织': ['同盟会']}\"))\n",
    "messages.extend(get_message(\"我曾在安徽芜湖市和上海浦东打工。\",\"{'地点': ['安徽芜湖市', '上海浦东']}\"))\n",
    "display(messages)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "30247e96-0f93-4c05-bc2d-8c836db05dd6",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "bdaf6e0f-fd17-40e1-b790-78593e382d93",
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict(text,temperature=0.01):\n",
    "    model.generation_config.temperature=temperature\n",
    "    response = model.chat(tokenizer, \n",
    "                          messages = messages+[{'role':'user','content':f'{text} -> '}])\n",
    "    return response\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d2dffa85-2932-4bee-b60d-965eedd88604",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"{'人名': ['杜甫', '李白']}\""
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predict('杜甫是李白的粉丝。') "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "659f9d62-34bb-4589-9bb5-f25090199775",
   "metadata": {},
   "source": [
    "我们拿一个开源的中文NER数据集来测试一下未经微调，仅仅使用7-shot-prompt的预训练模型的效果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "99ba497f-6959-4618-b4d2-d7da00e206af",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "import pandas as pd \n",
    "\n",
    "df = pd.read_pickle('dfner_13k.pkl')\n",
    "dfdata,dftest = train_test_split(df,test_size=300,random_state=42)\n",
    "dftrain,dfval = train_test_split(dfdata,test_size=200,random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "44fe3c7a-2857-478c-9041-42713e4b5b9b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d1a2cc92abff4e6e89c49fa083fb2796",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/300 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "preds = ['' for x in dftest['target']]\n",
    "for i in tqdm(range(len(preds))):\n",
    "    preds[i] = predict(dftest['text'].iloc[i])\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "d08cae8f-efd3-4e71-b76b-a31cfdc81de6",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "834d2b93-4080-43cd-934f-027ef01837bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "def toset(s):\n",
    "    try:\n",
    "        dic = eval(str(s))\n",
    "        res = []\n",
    "        for k,v in dic.items():\n",
    "            for x in v:\n",
    "                if x:\n",
    "                    res.append((k,x))\n",
    "        return set(res)\n",
    "    except Exception as err:\n",
    "        print(err)\n",
    "        return set()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "211babb9-2134-413a-ab1d-53399ca09429",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "precision = 0.4316109422492401\n",
      "recall = 0.45151033386327505\n",
      "f1_score = 0.44133644133644134\n"
     ]
    }
   ],
   "source": [
    "dftest['pred'] = [toset(x) for x in preds]\n",
    "dftest['gt'] = [toset(x) for x in dftest['target']]\n",
    "dftest['tp_cnt'] = [len(pred&gt) for pred,gt in zip(dftest['pred'],dftest['gt'])]\n",
    "dftest['pred_cnt'] = [len(x) for x in dftest['pred']]\n",
    "dftest['gt_cnt'] = [len(x) for x in dftest['gt']]\n",
    "\n",
    "precision = sum(dftest['tp_cnt'])/sum(dftest['pred_cnt'])\n",
    "print('precision = '+str(precision))\n",
    "\n",
    "recall = sum(dftest['tp_cnt'])/sum(dftest['gt_cnt'])\n",
    "print('recall = '+str(recall))\n",
    "\n",
    "f1 = 2*precision*recall/(precision+recall)\n",
    "print('f1_score = '+str(f1))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3e6f3ec-9a52-468d-a906-0e475c19e76e",
   "metadata": {},
   "source": [
    "微调前 f1_score为 0.44 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc98d03e-5d63-45d5-918e-19ca56eb6992",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "09be19b6-615f-4488-bd27-6dc08a015105",
   "metadata": {},
   "source": [
    "## 一，准备数据"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23d48441-441e-41c0-9442-7aeb8da003c3",
   "metadata": {},
   "source": [
    "我们仿照百川模型的 `model._build_chat_input` 方法来进行token编码，同时把需要学习的内容添加label."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d442edb-dcd5-46cf-91a9-3203bccbe9a4",
   "metadata": {},
   "source": [
    "### 1，token编码"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "9fcd2921-ca7d-43c6-b49b-e4e6a65ed37d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "\n",
    "#将messages编码成 token, 同时返回labels\n",
    "#注意baichuan-13b通过插入tokenizer.user_token_id和tokenizer.assistant_token_id 来区分用户和机器人会话内容\n",
    "\n",
    "# reference@ model._build_chat_input?\n",
    "def build_chat_input(messages, model=model,\n",
    "                     tokenizer=tokenizer, \n",
    "                     max_new_tokens: int=0):\n",
    "    max_new_tokens = max_new_tokens or model.generation_config.max_new_tokens\n",
    "    max_input_tokens = model.config.model_max_length - max_new_tokens\n",
    "    max_input_tokens = max(model.config.model_max_length // 2, max_input_tokens)\n",
    "    \n",
    "    total_input, round_input, total_label, round_label = [], [], [], []\n",
    "    \n",
    "    for i, message in enumerate(messages[::-1]):\n",
    "        content_tokens = tokenizer.encode(message['content'])\n",
    "        if message['role'] == 'user':\n",
    "            round_input = [model.generation_config.user_token_id] + content_tokens + round_input\n",
    "            round_label = [-100]+[-100 for _ in content_tokens]+ round_label\n",
    "            \n",
    "            if total_input and len(total_input) + len(round_input) > max_input_tokens:\n",
    "                break\n",
    "            else:\n",
    "                total_input = round_input + total_input\n",
    "                total_label = round_label + total_label\n",
    "                if len(total_input) >= max_input_tokens:\n",
    "                    break\n",
    "                else:\n",
    "                    round_input = []\n",
    "                    round_label = []\n",
    "                    \n",
    "        elif message['role'] == 'assistant':\n",
    "            round_input = [\n",
    "                model.generation_config.assistant_token_id\n",
    "            ] + content_tokens + [\n",
    "                model.generation_config.eos_token_id\n",
    "            ] + round_input\n",
    "            \n",
    "            if i==0: #仅对最后一轮的target进行学习\n",
    "                round_label = [\n",
    "                    -100\n",
    "                ] + content_tokens + [\n",
    "                    model.generation_config.eos_token_id\n",
    "                ]+ round_label\n",
    "            else:\n",
    "                round_label = [\n",
    "                    -100\n",
    "                ] + [-100 for _ in content_tokens] + [\n",
    "                    -100\n",
    "                ]+ round_label\n",
    "                \n",
    "        else:\n",
    "            raise ValueError(f\"message role not supported yet: {message['role']}\")\n",
    "            \n",
    "    total_input = total_input[-max_input_tokens:]  # truncate left\n",
    "    total_label = total_label[-max_input_tokens:]\n",
    "    \n",
    "    total_input.append(model.generation_config.assistant_token_id)\n",
    "    total_label.append(-100)\n",
    "    \n",
    "    return total_input,total_label\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c2d9290-53a7-4ae3-a1ca-d972512c72cb",
   "metadata": {},
   "source": [
    "### 2，做数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "b9a76500-01db-4c34-b2ba-6bc1c56233d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import Dataset,DataLoader \n",
    "from copy import deepcopy\n",
    "class MyDataset(Dataset):\n",
    "    def __init__(self,df,\n",
    "                 messages\n",
    "                ):\n",
    "        self.df = df \n",
    "        self.messages = messages\n",
    "        \n",
    "    def __len__(self):\n",
    "        return len(self.df)\n",
    "        \n",
    "    def get_samples(self,index):\n",
    "        samples = []\n",
    "        d = dict(self.df.iloc[index])\n",
    "        samples.append(d)\n",
    "        return samples\n",
    "    \n",
    "    def get_messages(self,index):\n",
    "        samples = self.get_samples(index)\n",
    "        messages = deepcopy(self.messages)\n",
    "        for i,d in enumerate(samples):\n",
    "\n",
    "            messages.append({'role':'user','content':d['text']+' -> '})\n",
    "            messages.append({'role':'assistant','content':str(d['target'])})\n",
    "        return messages\n",
    "        \n",
    "    def __getitem__(self,index):\n",
    "        messages = self.get_messages(index)\n",
    "        input_ids, labels = build_chat_input(messages)\n",
    "        return {'input_ids':input_ids,'labels':labels}\n",
    "\n",
    "    def show_sample(self,index):\n",
    "        samples = self.get_samples(index)\n",
    "        print(samples)\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "6cc385bf-0103-4720-a4bd-42744aa76f9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds_train = MyDataset(dftrain,messages)\n",
    "ds_val = MyDataset(dfval,messages)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "f19fe484-bc8b-44dd-8b26-d1a8714c0990",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'text': '包括中国在内的近30个国家的政府官员和经济专家参加了这次为期两天的研讨会。', 'target': {'地点': ['中国']}}]\n"
     ]
    }
   ],
   "source": [
    "ds_train.show_sample(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "d4b425fb-b9b7-4be6-bace-af6885843b50",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'input_ids': [195, 31106, 31710, 31313, 19529, 7254, 77, 32482, 31509, 31271, 31267, 3026, 20134, 31313, 72, 8021, 72, 2858, 12477, 31318, 31548, 31710, 31313, 19529, 72, 31404, 4201, 14013, 20983, 17965, 4021, 73, 5, 5, 7810, 31161, 2719, 31751, 31530, 77, 5, 5, 31222, 31333, 31209, 31222, 31641, 31263, 18706, 31203, 17510, 31230, 31291, 31447, 31763, 75, 31157, 11531, 1396, 31157, 31167, 31313, 5308, 29567, 31222, 31333, 15286, 31222, 31641, 31157, 5902, 966, 2858, 5308, 29567, 31291, 31447, 14507, 31277, 5, 2272, 72, 9437, 31169, 17271, 31456, 17966, 31195, 3366, 24749, 72, 31612, 31918, 31313, 1572, 25090, 31195, 3366, 31194, 31181, 73, 11531, 1396, 31157, 8021, 5308, 29567, 1572, 2286, 966, 3366, 14507, 31277, 5, 1572, 31161, 6583, 31200, 31291, 31240, 31190, 31349, 31424, 23942, 31200, 5760, 73, 11531, 1396, 31157, 8021, 5308, 29567, 1572, 31157, 5902, 966, 2858, 5308, 29567, 6583, 31200, 14507, 31277, 5, 5, 31488, 31209, 31214, 32193, 31271, 31267, 1696, 19529, 32482, 31509, 72, 17965, 14013, 20983, 73, 5, 5, 2719, 32473, 32783, 32253, 31799, 32433, 31453, 31681, 35076, 10209, 72, 7910, 31506, 31185, 31652, 32244, 33888, 32988, 11531, 31106, 196, 1396, 31155, 8021, 9077, 21575, 32473, 32783, 32253, 8504, 31277, 2, 195, 31106, 7674, 32246, 31195, 3659, 31200, 31291, 31635, 72, 9754, 18731, 31513, 31583, 31216, 32920, 75, 11531, 31106, 196, 1396, 31155, 2858, 9077, 21575, 3659, 31200, 31291, 31635, 8504, 31277, 2, 195, 31106, 10123, 31549, 31167, 32489, 20731, 31173, 27366, 6560, 31280, 32635, 31190, 26901, 11531, 31106, 196, 1396, 31155, 31167, 31313, 9077, 21575, 32489, 20731, 26082, 1346, 31218, 31313, 9077, 21575, 27366, 26082, 1346, 2858, 9077, 21575, 31280, 32635, 31190, 8504, 31277, 2, 195, 6323, 27707, 9688, 34061, 31893, 31236, 31188, 4512, 33074, 31440, 22449, 73, 11531, 31106, 196, 1396, 31155, 8021, 9077, 21575, 9688, 34061, 31893, 31236, 2745, 1346, 4512, 33074, 31440, 8504, 31277, 2, 195, 31106, 4296, 1572, 31173, 8342, 31554, 55, 52, 31179, 19769, 3192, 28049, 31188, 2887, 5593, 25202, 7674, 31178, 31303, 31444, 9179, 24910, 31190, 73, 11531, 31106, 196, 1396, 31155, 8021, 9077, 21575, 1572, 8504, 31277, 2, 196], 'labels': [-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 1396, 31155, 8021, 9077, 21575, 1572, 8504, 31277, 2, -100]}\n"
     ]
    }
   ],
   "source": [
    "print(ds_train[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23b928e3-e464-4740-863c-dc3c651f3efa",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "bc80b2ee-7d70-41a7-8cf7-2bb5fdfff151",
   "metadata": {},
   "source": [
    "### 3，创建管道"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "e43aaccc-863e-4794-8cb5-683d5a727ada",
   "metadata": {},
   "outputs": [],
   "source": [
    "def data_collator(examples: list):\n",
    "    len_ids = [len(example[\"input_ids\"]) for example in examples]\n",
    "    longest = max(len_ids) #之后按照batch中最长的input_ids进行padding\n",
    "    \n",
    "    input_ids = []\n",
    "    labels_list = []\n",
    "    \n",
    "    for length, example in sorted(zip(len_ids, examples), key=lambda x: -x[0]):\n",
    "        ids = example[\"input_ids\"]\n",
    "        labs = example[\"labels\"]\n",
    "        \n",
    "        ids = ids + [tokenizer.pad_token_id] * (longest - length)\n",
    "        labs = labs + [-100] * (longest - length)\n",
    "        \n",
    "        input_ids.append(torch.LongTensor(ids))\n",
    "        labels_list.append(torch.LongTensor(labs))\n",
    "          \n",
    "    input_ids = torch.stack(input_ids)\n",
    "    labels = torch.stack(labels_list)\n",
    "    return {\n",
    "        \"input_ids\": input_ids,\n",
    "        \"labels\": labels,\n",
    "    }\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "698aa807-e05a-42a8-b392-747982363075",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "dl_train = torch.utils.data.DataLoader(ds_train,num_workers=2,batch_size=1,\n",
    "                                       pin_memory=True,shuffle=True,\n",
    "                                       collate_fn = data_collator)\n",
    "\n",
    "dl_val = torch.utils.data.DataLoader(ds_val,num_workers=2,batch_size=1,\n",
    "                                    pin_memory=True,shuffle=False,\n",
    "                                     collate_fn = data_collator)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "db52d785-10de-4465-8fc2-9eaccaf3387d",
   "metadata": {},
   "outputs": [],
   "source": [
    "for batch in dl_train:\n",
    "    break "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "35002105-b8e8-4d48-a8a2-cbd4ceaa53ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "#试跑一个batch\n",
    "out = model(**batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "c721189c-6443-4e10-b923-327a0beba2fd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(1.3135, dtype=torch.float16, grad_fn=<ToCopyBackward0>)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out.loss "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "7cac7d4a-a37b-4c30-8a7d-be552d8e67ca",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5000"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(dl_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "2eb0f563-61b0-4576-a3b1-08aa4b62e52f",
   "metadata": {},
   "outputs": [],
   "source": [
    "#采样300个batch作为一个epoch，便于较快验证\n",
    "dl_train.size = 300"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37fd7e26-985a-4392-ba0d-acb254064677",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "5aa1deea-3fc4-4051-80e9-89de248d3107",
   "metadata": {},
   "source": [
    "## 二，定义模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "644837f5-bbc4-4f4f-a5f0-1f763e36f373",
   "metadata": {},
   "source": [
    "下面我们将使用QLoRA(实际上用的是量化的AdaLoRA）算法来微调Baichuan-13b模型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "b7dbbece-2e50-4166-9e43-ea35d04af856",
   "metadata": {},
   "outputs": [],
   "source": [
    "from peft import get_peft_config, get_peft_model, TaskType\n",
    "model.supports_gradient_checkpointing = True  #\n",
    "model.gradient_checkpointing_enable()\n",
    "model.enable_input_require_grads()\n",
    "\n",
    "model.config.use_cache = False  # silence the warnings. Please re-enable for inference!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "fc66d6a4-77ff-490c-a3c0-913a9316eb13",
   "metadata": {},
   "outputs": [],
   "source": [
    "import bitsandbytes as bnb \n",
    "def find_all_linear_names(model):\n",
    "    \"\"\"\n",
    "    找出所有全连接层，为所有全连接添加adapter\n",
    "    \"\"\"\n",
    "    cls = bnb.nn.Linear4bit\n",
    "    lora_module_names = set()\n",
    "    for name, module in model.named_modules():\n",
    "        if isinstance(module, cls):\n",
    "            names = name.split('.')\n",
    "            lora_module_names.add(names[0] if len(names) == 1 else names[-1])\n",
    "\n",
    "    if 'lm_head' in lora_module_names:  # needed for 16-bit\n",
    "        lora_module_names.remove('lm_head')\n",
    "    return list(lora_module_names)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "587a53eb-ba75-40a6-83e6-1ad2684738f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from peft import prepare_model_for_kbit_training \n",
    "model = prepare_model_for_kbit_training(model)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "ffbd01ad-2667-4c38-8ce7-62ce293e759e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['down_proj', 'gate_proj', 'W_pack', 'o_proj', 'up_proj']\n"
     ]
    }
   ],
   "source": [
    "lora_modules = find_all_linear_names(model)\n",
    "print(lora_modules) \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "e5b68e33-3c18-4044-9554-c453b17d4fef",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "trainable params: 41,843,040 || all params: 7,002,181,160 || trainable%: 0.5975715144165165\n"
     ]
    }
   ],
   "source": [
    "from peft import AdaLoraConfig\n",
    "peft_config = AdaLoraConfig(\n",
    "    task_type=TaskType.CAUSAL_LM, inference_mode=False,\n",
    "    r=16,\n",
    "    lora_alpha=16, lora_dropout=0.05,\n",
    "    target_modules= lora_modules\n",
    ")\n",
    "\n",
    "peft_model = get_peft_model(model, peft_config)\n",
    "\n",
    "peft_model.is_parallelizable = True\n",
    "peft_model.model_parallel = True\n",
    "peft_model.print_trainable_parameters()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "372d96e5-f5c4-4a21-a09b-825df8563399",
   "metadata": {},
   "outputs": [],
   "source": [
    "out = peft_model.forward(**batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "7b1c036a-c466-45a8-816e-e93da6810c87",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(1.3118, grad_fn=<ToCopyBackward0>)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74ddcd08-851b-42e0-a1ac-db530e728aee",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "2ad2444d-2c9c-4048-958c-ae48d9590dfd",
   "metadata": {},
   "source": [
    "## 三，训练模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "9317cd85-b886-4153-af53-42c83921525a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchkeras import KerasModel \n",
    "from accelerate import Accelerator \n",
    "\n",
    "class StepRunner:\n",
    "    def __init__(self, net, loss_fn, accelerator=None, stage = \"train\", metrics_dict = None, \n",
    "                 optimizer = None, lr_scheduler = None\n",
    "                 ):\n",
    "        self.net,self.loss_fn,self.metrics_dict,self.stage = net,loss_fn,metrics_dict,stage\n",
    "        self.optimizer,self.lr_scheduler = optimizer,lr_scheduler\n",
    "        self.accelerator = accelerator if accelerator is not None else Accelerator() \n",
    "        if self.stage=='train':\n",
    "            self.net.train() \n",
    "        else:\n",
    "            self.net.eval()\n",
    "    \n",
    "    def __call__(self, batch):\n",
    "        \n",
    "        #loss\n",
    "        with self.accelerator.autocast():\n",
    "            loss = self.net.forward(**batch)[0]\n",
    "\n",
    "        #backward()\n",
    "        if self.optimizer is not None and self.stage==\"train\":\n",
    "            self.accelerator.backward(loss)\n",
    "            if self.accelerator.sync_gradients:\n",
    "                self.accelerator.clip_grad_norm_(self.net.parameters(), 1.0)\n",
    "            self.optimizer.step()\n",
    "            if self.lr_scheduler is not None:\n",
    "                self.lr_scheduler.step()\n",
    "            self.optimizer.zero_grad()\n",
    "            \n",
    "        all_loss = self.accelerator.gather(loss).sum()\n",
    "        \n",
    "        #losses (or plain metrics that can be averaged)\n",
    "        step_losses = {self.stage+\"_loss\":all_loss.item()}\n",
    "        \n",
    "        #metrics (stateful metrics)\n",
    "        step_metrics = {}\n",
    "        \n",
    "        if self.stage==\"train\":\n",
    "            if self.optimizer is not None:\n",
    "                step_metrics['lr'] = self.optimizer.state_dict()['param_groups'][0]['lr']\n",
    "            else:\n",
    "                step_metrics['lr'] = 0.0\n",
    "        return step_losses,step_metrics\n",
    "    \n",
    "KerasModel.StepRunner = StepRunner \n",
    "\n",
    "#仅仅保存QLora可训练参数\n",
    "def save_ckpt(self, ckpt_path='checkpoint', accelerator = None):\n",
    "    unwrap_net = accelerator.unwrap_model(self.net)\n",
    "    unwrap_net.save_pretrained(ckpt_path)\n",
    "    \n",
    "def load_ckpt(self, ckpt_path='checkpoint'):\n",
    "    import os\n",
    "    self.net.load_state_dict(\n",
    "        torch.load(os.path.join(ckpt_path,'adapter_model.bin')),strict =False)\n",
    "    self.from_scratch = False\n",
    "    \n",
    "KerasModel.save_ckpt = save_ckpt \n",
    "KerasModel.load_ckpt = load_ckpt \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "353a96d4-0fef-419a-a345-f5a5342ab8cc",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "cbed7bbe-10a7-4132-8d02-ead60b4d59ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = bnb.optim.adamw.AdamW(peft_model.parameters(),\n",
    "                                  lr=6e-05,is_paged=True)  #'paged_adamw'\n",
    "keras_model = KerasModel(peft_model,loss_fn =None,\n",
    "        optimizer=optimizer) \n",
    "ckpt_path = 'baichuan13b_ner'\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "131ba617-d6e6-4bf7-9259-daaa113ada8f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;31m<<<<<< ⚡️ cuda is used >>>>>>\u001b[0m\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiEAAAGJCAYAAABcsOOZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAB46UlEQVR4nO3deVxUVf8H8M8wyIDsyC4oKK6kUC5kRmniWqaiuWVu7ZpppKY9qZk9UZaG5pb1K63ULMR6yrKUxEhxyXLJ1MQgwdhEFkUFmTm/P64zMjDLBQYG5PN+vUaYO2fOPXeucL+c5XsVQggBIiIionpmY+0GEBERUdPEIISIiIisgkEIERERWQWDECIiIrIKBiFERERkFQxCiIiIyCoYhBAREZFVMAghIiIiq2AQQkRERFbBIITq3KuvvgqFQoGLFy9auyn1Jj09HQqFAhs2bKjT91Djs3TpUnTs2BEajcbaTak3DfF3wLx58xAREWHtZjR5DELotvXGG2/gq6++snYzqJLCwkI89dRT8PLygqOjI/r27YvffvtN9vtPnTqFQYMGwcnJCR4eHnjssceQl5dXpVxWVhaeeuopBAcHw8HBAW3btkVMTAzy8/P1yn3wwQe4//774ePjA5VKheDgYEyZMgXp6ekm2/HLL79AoVBU6+JaXFyMt956Cy+99BJsbMz/+r1+/Tpu3Lghq+7b2fbt2zFw4ED4+/tDpVIhICAAo0aNwh9//GHyfefOnYO9vT0UCgV+/fVXvddmzZqFY8eO4X//+19dNp3MYBBCty0GIQ2PRqPBgw8+iM2bN+O5557D0qVLkZubiz59+uDs2bNm35+ZmYn77rsPqampeOONNzB79mzs2LED/fv3R1lZma7clStX0KtXL2zfvh0TJ07Ee++9hyFDhmDVqlWIiorS64X4/fffERwcjLlz52Lt2rWYMGECvv/+e/To0QP//vuv0eOYMWMGHB0dq3X8H330EcrLyzFu3DijZQ4fPozHHnsMPj4+cHBwgEqlQmBgIJ5//nmkpqZWa3+3ixMnTsDd3R0zZ87EmjVr8Oyzz+L3339Hz549cezYMaPve+GFF2Bra2vwNV9fXwwbNgzvvPNOXTWb5BBEdWzRokUCgMjLy6vX/To6OopJkybV6z610tLSBADx8ccf1+l7GputW7cKAOLLL7/UbcvNzRVubm5i3LhxZt//7LPPCgcHB/HPP//otu3atUsAEO+//75u26ZNmwQA8e233+q9f+HChQKA+O2330zu59dffxUARGxsrMHX165dK1q0aCFmzpxZrf/bXbt2FRMmTDD42o0bN8T06dOFQqEQkZGR4p133hHffPON2LZtm3jjjTdEeHi4sLe3F6tWrZK1r4akLn4HZGdnC1tbW/H0008bfH3nzp3Czs5OvPLKKwKAOHz4cJUy8fHxQqFQiHPnzlmsXVQ97AmhenPx4kWMHj0aLi4uaNGiBWbOnInr169XKffZZ5+hW7ducHBwgIeHB8aOHYuMjAy9MmfPnsXIkSPh6+sLe3t7BAQEYOzYsSgqKgIAKBQKlJSUYOPGjbou88mTJxtsV05ODmxtbbF48eIqr505cwYKhQKrVq0CAFy6dAmzZ89Gly5d4OTkBBcXFwwePNjkX2O19dNPPyEyMhKOjo5wc3PDsGHDcOrUKb0yly9fxqxZsxAUFASVSgVvb2/0799fb5jD3GdWH+Lj4+Hj44Po6GjdNi8vL4wePRpff/01SktLTb5/27ZteOihh9CqVSvdtqioKLRv3x5ffPGFbltxcTEAwMfHR+/9fn5+AAAHBweT+wkKCgIgDR1VdunSJbzyyit47bXX4ObmZrKeitLS0nD8+HFERUUZfH3KlCnYvHkzvvvuO/z888948cUX8dBDDyE6Ohrz58/H77//jnXr1mH27NlYt25dlfdfuHABU6dO1Q0rhYaG4qOPPtIrk5SUBIVCga1bt+Lll1+Gr68vHB0d8fDDD1f5GQOAL7/8Uvez6OnpiQkTJuDChQtVyp0+fRqjR4+Gl5cXHBwc0KFDB/znP/+pUq6wsBCTJ0+Gm5sbXF1dMWXKFFy9elXuR6jH29sbzZs3N3iObty4gZkzZ2LmzJlo27at0Tq05+Lrr7+uURvIAqwdBdHtT/tXUJcuXcTQoUPFqlWrxIQJEwQA8dhjj+mVff3114VCoRBjxowRa9asEYsXLxaenp4iKChIFBQUCCGEKC0tFcHBwcLf31+8/vrr4sMPPxSLFy8WPXr0EOnp6UIIIT799FOhUqlEZGSk+PTTT8Wnn34q9u/fb7SNDzzwgOjcuXOV7YsXLxZKpVJkZ2cLIYQ4fPiwaNu2rZg3b554//33xWuvvSZatmwpXF1dxYULF3Tvs1RPyK5du4Stra1o3769WLp0qe7zcHd3F2lpabpy48ePF3Z2diImJkZ8+OGH4q233hJDhw4Vn332mezPzJiSkhKRl5dn9nHp0iWzxxgSEiIGDx5cZfuHH34oAIjjx48bfW9mZqYAIN56660qr02YMEF4eHjonp88eVLY2NiIe+65R6SkpIiMjAyxY8cOERAQIIYPH26w/osXL4qcnBxx+PBhMXToUAFA/Pjjj1XKTZs2TYSGhory8vJq/YX/2WefGT3GTz75RDg6Ooo//vhDt02j0YjLly/rnufl5Qm1Wi2+/fZb4eDgoHfesrOzRUBAgAgMDBSvvfaaWLt2rXj44YcFAPHuu+/qyu3Zs0f3s9i1a1exfPlyMW/ePGFvby/at28vrl69qiv78ccfCwCiR48e4t133xXz5s0TDg4Oej+LQghx7Ngx4eLiIlq0aCHmz58v3n//fTF37lzRpUsXXRnt53TnnXeK6OhosWbNGvHEE08IAGLu3LlmPzutgoICkZubK44fPy6mTp0qAIj169dXKbd06VLh7e0tioqKdMdhqCdECOn/5MiRI2W3gSyLQQjVOe0voIcfflhv+7Rp0wQAcezYMSGEEOnp6UKpVIr//ve/euVOnDghbG1tddt///33Kl36hlRnOOb9998XAMSJEyf0tnfu3Fk88MADuufXr18XarVar0xaWppQqVTitdde09tmiSAkPDxceHt7i/z8fN22Y8eOCRsbGzFx4kTdNldXVzF9+nSjdcv9zAzRnj9zj9atW5uty9HRUUydOrXK9h07dggAYufOnUbfe/jwYQFAfPLJJ1VemzNnjgAgrl+/rtv24YcfCjc3N702Tpo0Sdy4ccNg/SqVSleuRYsWYuXKlVXKHDt2TCiVSvHDDz8IIao3zKAdFqgYWAghBRvBwcEiLi5Ot+3rr78W/v7+AoBo1aqV+OGHHwQAXeA5YsQI8fLLL+vKP/7448LPz09cvHhRr+6xY8cKV1dXXXChDUJatmwpiouLdeW++OILAUCsWLFCCCFEWVmZ8Pb2FnfccYe4du2arty3334rAIiFCxfqtt13333C2dlZb4hMe1xa2s+p8rkfMWKEaNGihdnPTqtDhw66c+Tk5CReeeWVKj+PWVlZwtnZWTc8Zy4IGTBggOjUqZPsNpBlcTiG6s306dP1ns+YMQMA8N133wEAEhISoNFoMHr0aFy8eFH38PX1Rbt27bBnzx4AgKurKwDghx9+qHFXbmXR0dGwtbXF1q1bddv++OMP/PnnnxgzZoxum0ql0q1qUKvVyM/Ph5OTEzp06FCtFR5yZGVl4ejRo5g8eTI8PDx027t27Yr+/fvrPjcAcHNzw8GDB41OpKzNZzZx4kTs2rXL7GPTpk1m67p27RpUKlWV7fb29rrXTb0XgOz3t2zZEj179kRcXBy2b9+OmJgYbNq0CfPmzTNY//fff4/vvvsOy5YtQ6tWrVBSUlKlzPPPP4/BgwdjwIABJo7SsPz8fNja2sLJyUlv+5EjR5Cbm4vHH38cgDSsMm7cOPTs2RPbtm3DCy+8gKlTp+q9Z/jw4UhKSgIACCGwbds2DB06FEIIvZ+dgQMHoqioqMr/zYkTJ8LZ2Vn3fNSoUfDz89P9n/r111+Rm5uLadOm6T5bAHjwwQfRsWNH7NixAwCQl5eHn3/+GVOnTtUbIgOkIdHKnnnmGb3nkZGRyM/P1w2fmfPxxx9j586dWLNmDTp16oRr165BrVbrlXnppZfQpk0bPPHEE7LqdHd3b1BLh5saw9OGiepAu3bt9J63bdsWNjY2uqWQZ8+ehRCiSjmtZs2aAQCCg4MRExOD5cuXY9OmTYiMjMTDDz+MCRMm6C621eXp6Yl+/frhiy++wJIlSwAAW7duha2trd78BY1GgxUrVmDNmjVIS0vT+wXYokWLGu3bmH/++QcA0KFDhyqvderUCT/88ANKSkrg6OiIpUuXYtKkSQgMDES3bt0wZMgQTJw4EW3atAFQu8+sTZs2unpqy8HBweC8D+3cIFNzNbSvyXn/vn378NBDD+HAgQPo3r07AOnC7eLigsWLF2Pq1Kno3LmzXh19+/YFAAwePBjDhg3DHXfcAScnJzz33HMApP8P+/fvN7sstLqOHDmC7t2764KTTZs2oWXLloiPj4dSqQQgBZlTpkzRvcfHx0e3LDkvLw+FhYVYv3491q9fb3Afubm5es8r/4wpFAqEhITofhZN/d/r2LEjfvnlFwDA33//DQC44447ZB1r5UDF3d0dAFBQUAAXFxez7+/Vq5fu+7Fjx6JTp04AoFvhcuDAAXz66adITEyUtQQakII4QwET1Q/2hJDVVP7B12g0UCgU2Llzp8G/tN9//31d2WXLluH48eN4+eWXce3aNTz//PMIDQ1FZmZmjdszduxY/PXXXzh69CgA4IsvvkC/fv3g6empK/PGG28gJiYG9913Hz777DP88MMP2LVrF0JDQ62afGr06NH4+++/8d5778Hf3x9vv/02QkND8f333+vK1PQzu3LlCrKzs80+DOXqqMzPzw9ZWVlVtmu3+fv7m3xvxbKV3+/h4aHrJXn//ffh4+OjC0C0Hn74YQghsH//fpPtbNu2Le6880693p05c+bgkUcegZ2dHdLT05Genq6bFJmRkWG0F0qrRYsWKC8vx+XLl/W25+fn6x13eno67rzzTl0AAgA9e/bUe09GRoYu6NX+v5swYYLRXqrevXubbFt9qXhMFQkhql2Xu7s7HnjgAb1zNHfuXERGRiI4OFh3jrS9HFlZWTh//nyVegoKCvR+xql+sSeE6s3Zs2cRHByse56amgqNRqNbidC2bVsIIRAcHIz27dubra9Lly7o0qULXnnlFezfvx+9e/fGunXr8PrrrwMw3B1syvDhw/H000/rhmT++usvzJ8/X69MfHw8+vbti//7v//T215YWGjxX2StW7cGIK3Qqez06dPw9PTUy1Ph5+eHadOmYdq0acjNzcVdd92F//73vxg8eLCujLnPzJB33nnH4MohQ+01l+ArPDwcycnJ0Gg0en+pHjx4EM2bNzd53lu2bAkvL68qSacA4NChQwgPD9c9z8nJqdJND0CX+Ku8vNzM0UhDOxV7XTIyMrB582Zs3ry5Stm77roLYWFhugDWkI4dOwKQVsl07dpVt93FxUVvhZKvry8OHTqk915tjwMgXbD/7//+T7eyw8vLC87OzlCr1UZX3lRWOSeLEAKpqam6dlX8v/fAAw/olT1z5ozudW0PmaV7h+S6du2a3md3/vx5/PPPP3q/Z7QefvhhuLq6VllNk5aWhrCwsLpuKhnBnhCqN6tXr9Z7/t577wGA7iIZHR0NpVKJxYsXV/nLSAihy3RZXFxc5SLSpUsX2NjY6F00HB0dDS7fM8bNzQ0DBw7EF198gc8//xx2dnYYPny4XhmlUlmlbV9++aXBZYu15efnh/DwcGzcuFHvOP744w/8+OOPGDJkCABpbkrlZbbe3t7w9/fXfR5yPzNDLDknZNSoUcjJyUFCQoJu28WLF/Hll19i6NChevM9zp07h3Pnzum9f+TIkfj222/1lpMmJibir7/+wiOPPKLb1r59e+Tk5OjmTWht2bIFAHDnnXcCkIKRgoKCKu08dOgQTpw4odeTsn379ioP7XyhTz75BO+++67JY9cOJVQOojp16oTDhw/rejSGDRuG33//HQsXLsTff/+N5ORkzJkzB4CUWG3kyJHIzMzEzJkzAUj/J0eOHIlt27YZDAYM9VB98sknej0y8fHxyMrK0v0sdu/eHd7e3li3bp3e/4/vv/8ep06dwoMPPghACoDuu+8+fPTRR1V6GWrSu2FM5eEkQOoxSkxM1DtH69evr3KOtHPP3nnnnSr/R4uKinDu3Dncc889FmsrVZNVpsNSk1J5ie7q1at1S3THjx+vVzY2NlYAEPfcc49YunSpWLt2rZg7d65o166dePvtt4UQQmzfvl20bNlSzJo1S6xZs0asXLlS9OjRQzRr1kykpKTo6hoyZIhwdHQUy5YtE1u2bBEHDhww21btMkpnZ2cxdOjQKq9rk11NnjxZrF+/XsyYMUN4eHiINm3aiPvvv19XztJLdDt27Cjefvtt8dprrwkvLy/h7u4u/v77byGEtGxRuxJo+fLlYv369WL06NECgFi2bFm1PrO6Vl5eLu6++27h5OQkFi9eLFavXi1CQ0OFs7OzOH36tF7Z1q1bV1lxc/78edGiRQvRtm1bsXLlSvHGG28Id3d30aVLF72VMadPnxaOjo7CyclJzJ8/X6xbt06MGzdOABD9+/fXldN+dlOnThXLli0T69atE9OnTxfNmzcXHh4e4q+//jJ5PNVNwnXHHXdUScp2/fp14erqKrZv367b9sYbbwgbGxsBQNja2ooVK1boVoUMGDBAd+61srOzRevWrUXz5s3FzJkzxfvvvy9iY2PFI488Itzd3XXlKi/R1S69tbe3FyEhIaKkpERXVruqJCIiQsTFxYn58+eL5s2bV1mie/ToUeHk5KRbort+/Xrx8ssvi7CwMLOfk3YfFZebG+Lt7S3GjRsn3nrrLbF+/XoxZ84c4eHhIezt7cW+fftMvtfU6pj4+HgBQKSmppqsg+oOgxCqc9pfQH/++acYNWqUcHZ2Fu7u7uK5557TW/6ntW3bNnHvvfcKR0dH4ejoKDp27CimT58uzpw5I4QQ4u+//xZTp04Vbdu2Ffb29sLDw0P07dtX7N69W6+e06dPi/vuu084ODjolmeaU1xcrCuvzbFR0fXr18WLL74o/Pz8hIODg+jdu7dISUkR999/f50EIUIIsXv3btG7d2/h4OAgXFxcxNChQ8Wff/6pe720tFTMmTNHhIWFCWdnZ+Ho6CjCwsLEmjVrdGXkfmb14dKlS+Lxxx8XLVq0EM2bNxf333+/wQuEoSBECCH++OMPMWDAANG8eXPh5uYmHn30UV0el4pOnz4tRo0aJQIDA0WzZs1E69atxezZs/UutKWlpWLmzJmia9euwsXFRVfu8ccfN3thFKL6Qcjy5cuFk5OTXj4ObT1t2rTRy7Vy4cIF8fPPP+uO7ZdffhG5ublG687JyRHTp0/XHa+vr6/o16+fXh4NbRCyZcsWMX/+fOHt7S0cHBzEgw8+WGWJrRBShts777xTqFQq4eHhIR599FGRmZlZpdwff/whRowYIdzc3IS9vb3o0KGDWLBggdnPSW4QsmjRItG9e3fh7u4ubG1thb+/vxg7dqzJvDKV92Ho/9iYMWPEvffea7YOqjsKISzYZ0ZEREYVFRWhTZs2WLp0qW5JLiCt7unduzeUSiW+/vpr3STcyuLj4zFixAijEzzNSUpKQt++ffHll19i1KhRNarjdpGdnY3g4GB8/vnnGDZsmLWb02RxTggRUT1xdXXF3Llz8fbbb+utprK3t8d3330HhUKBDh064KWXXsLPP/+Mf/75B6dPn8Ynn3yCXr16YdKkSRbPR9NUxcXFoUuXLgxArIw9IUR1rKysDJcuXTJZxtXV1ez9TOj2V1ZWhlWrVmHVqlVIS0vTbbe3t8eIESOwePFio3l05GBPCDU0XKJLVMf279+vS4RlzMcff2z0BnvUdNjZ2SEmJgYxMTFIT0/HhQsXYG9vj06dOqF58+bWbh6RxbEnhKiOFRQU4MiRIybLhIaGGp0HQER0u2oQQcjq1avx9ttvIzs7G2FhYXjvvfeqZAg05PPPP8e4ceMwbNgwfPXVV7rtQggsWrQIH3zwAQoLC9G7d2+sXbu2Vt2YREREZFlWn5i6detWxMTEYNGiRfjtt98QFhaGgQMHGkxOU1F6ejpmz56NyMjIKq8tXboUK1euxLp163Dw4EE4Ojpi4MCBuvtLEBERkfVZvSckIiICPXr0wKpVqwBI90EIDAzEjBkzjN7tUq1W47777sPUqVORnJyMwsJCXU+IEAL+/v548cUXMXv2bADSsjgfHx9s2LABY8eONdsmjUaDf//9F87OzryxERERUTUIIXD58mX4+/ubvZGgVSemlpWV4ciRI3r357CxsUFUVBRSUlKMvu+1116Dt7c3Hn/8cSQnJ+u9lpaWhuzsbL17KLi6uiIiIgIpKSkGg5DS0lK91MQXLlyocodNIiIiki8jIwMBAQEmy1g1CLl48SLUajV8fHz0tvv4+OD06dMG3/PLL7/g//7v/4zeKCo7O1tXR+U6ta9VFhsba/AGXRkZGbJuL01ERESS4uJiBAYGwtnZ2WzZRrVE9/Lly3jsscfwwQcfWPSOpfPnz0dMTIzuufYDdHFxYRBCRERUA3KmM1g1CPH09IRSqUROTo7e9pycHPj6+lYpf+7cOaSnp2Po0KG6bdqsg7a2tjhz5ozufTk5OXpLHnNycvRu9V2RSqXSu3snERER1T2rro6xs7NDt27dkJiYqNum0WiQmJiou+11RR07dsSJEydw9OhR3ePhhx9G3759cfToUQQGBiI4OBi+vr56dRYXF+PgwYMG6yQiIiLrsPpwTExMDCZNmoTu3bujZ8+eiIuLQ0lJCaZMmQIAmDhxIlq2bInY2FjY29vjjjvu0Hu/m5sbAOhtnzVrFl5//XW0a9cOwcHBWLBgAfz9/TF8+PD6OiwiIiIyw+pByJgxY5CXl4eFCxciOzsb4eHh2Llzp25i6fnz580u8als7ty5KCkpwVNPPYXCwkLce++92LlzJ+zt7eviEIiISCYhBMrLy6FWq63dFKohpVIJW1tbi6SwsHqekIaouLgYrq6uKCoq4sRUIiILKSsrQ1ZWFq5evWrtplAtNW/eHH5+frCzs6vyWnWuoVbvCSEiotufRqNBWloalEol/P39YWdnx2SQjZAQAmVlZcjLy0NaWhratWtX7dGKihiE1AO1GkhOBrKyAD8/IDISUCqt3SoiovpTVlamy4jNOwI3bg4ODmjWrBn++ecflJWV1WqqA4OQOpaQAMycCWRm3toWEACsWAFER1uvXURE1lCbv5qp4bDUeeT/hjqUkACMGqUfgADAhQvS9oQE67SLiIioIWAQUkfUaqkHxNC0X+22WbOkckRERE0Rg5A6kpxctQekIiGAjAypHBERyaNWA0lJwJYt0tfG9odcUFAQ4uLiLFJXUlISFAoFCgsLLVKfNXBOSB3JyrJsOSKips5ac+z69OmD8PBwiwQPhw8fhqOjY+0bdZtgT0gdqXDbGouUIyJqyhryHDttAjY5vLy8uDqoAgYhdSQyUorQjS2DVyiAwECpHBFRU1ZSYvxx/bq8OXYzZwJXrpivt7omT56MvXv3YsWKFVAoFFAoFNiwYQMUCgW+//57dOvWDSqVCr/88gvOnTuHYcOGwcfHB05OTujRowd2796tV1/l4RiFQoEPP/wQI0aMQPPmzdGuXTv873//q35Db9q2bRtCQ0OhUqkQFBSEZcuW6b2+Zs0atGvXDvb29vDx8cGoUaN0r8XHx6NLly5wcHBAixYtEBUVhZKafGjVwCCkjiiVUhchUDUQ0T6Pi2O+ECIiJyfjj5Ej5c2xy8wE7r1Xf3tQUNX6qmvFihXo1asXnnzySWRlZSErKwuBgYEAgHnz5uHNN9/EqVOn0LVrV1y5cgVDhgxBYmIifv/9dwwaNAhDhw7F+fPnTe5j8eLFGD16NI4fP44hQ4bg0UcfxaVLl6rd1iNHjmD06NEYO3YsTpw4gVdffRULFizAhg0bAAC//vornn/+ebz22ms4c+YMdu7cifvuuw8AkJWVhXHjxmHq1Kk4deoUkpKSEB0djTpPqi6oiqKiIgFAFBUV1bqubduECAgQQvoxkR6BgdJ2IqKm4tq1a+LPP/8U165dq/Jaxd+PlR9DhgixebPpMtpHp0769Xp6Vi1TE/fff7+YOXOm7vmePXsEAPHVV1+ZfW9oaKh47733dM9bt24t3n333QrHDvHKK6/onl+5ckUAEN9//73ZurXtKCgoEEIIMX78eNG/f3+9MnPmzBGdO3cWQgixbds24eLiIoqLi6vUdeTIEQFApKenm92vEKbPZ3WuoewJqWPR0UB6OvDyy9LzO+8E0tKYqIyISOvKFeOPbdvkz51791395+npVeuzpO7du+s9v3LlCmbPno1OnTrBzc0NTk5OOHXqlNmekK5du+q+d3R0hIuLC3Jzc6vdnlOnTqF3795623r37o2zZ89CrVajf//+aN26Ndq0aYPHHnsMmzZt0t3HJywsDP369UOXLl3wyCOP4IMPPkBBQUG121BdDELqgVIJ9O0rfV9WxiEYIqKKHB2NP+zt5c+xi4oyX69l261f4ezZs7F9+3a88cYbSE5OxtGjR9GlSxeUlZWZrKdZs2Z6zxUKBTQajWUbC8DZ2Rm//fYbtmzZAj8/PyxcuBBhYWEoLCyEUqnErl278P3336Nz585477330KFDB6SlpVm8HRUxCKknoaHSHJH//tfaLSEialysPcfOzs4OahkJSfbt24fJkydjxIgR6NKlC3x9fZGenl43jTKgU6dO2LdvX5U2tW/fHsqbH46trS2ioqKwdOlSHD9+HOnp6fjpp58ASMFP7969sXjxYvz++++ws7PD9u3b67TNzBNST/z8gOeft3YriIgap+hoID7ecJ6QuLi6HeIOCgrCwYMHkZ6eDicnJ6O9FO3atUNCQgKGDh0KhUKBBQsW1EmPhjEvvvgievTogSVLlmDMmDFISUnBqlWrsGbNGgDAt99+i7///hv33Xcf3N3d8d1330Gj0aBDhw44ePAgEhMTMWDAAHh7e+PgwYPIy8tDp06d6rTN7AkhIqJGQTvHbs8eYPNm6Wt9zLGbPXs2lEolOnfuDC8vL6NzPJYvXw53d3fcc889GDp0KAYOHIi77rqrbhtXwV133YUvvvgCn3/+Oe644w4sXLgQr732GiZPngwAcHNzQ0JCAh544AF06tQJ69atw5YtWxAaGgoXFxf8/PPPGDJkCNq3b49XXnkFy5Ytw+DBg+u0zQoh6nr9TeNTXFwMV1dXFBUVwcXFxWL1HjkCnD8P9O4NeHtbrFoiogbv+vXrSEtLQ3BwcK1u/U4Ng6nzWZ1rKHtC6tETT0gR+5Ej1m4JERGR9TEIqUe+vtLX7GzrtoOIiBq+Z555Bk5OTgYfzzzzjLWbZxGcmFqPfHykrzk51m0HERE1fK+99hpmz55t8DVLThWwJgYh9Yg9IUREJJe3tze8b/MJhByOqUfsCSEiIrqFQUg9Yk8IERHRLQxC6hF7QoiIiG7hnJB61LmzlHq4dWtrt4SIiMj6GITUI19fpm4nIiLS4nAMERE1GmohkFRQgC05OUgqKIC6EST9DgoKQlxcnKyyCoUCX331VZ22pyFhT0g9O3IE+OcfKXW7do4IERGZl5CXh5mpqcgsLdVtC1CpsCIkBNFeXlZsGdUUe0Lq2VNPASNHAr/+au2WEBE1Hgl5eRh18qReAAIAF0pLMerkSSTk5VmpZVQbDELqGZfpEhFJhBAoUavNPorLy/H82bMwNPCi3TYzNRXF5eWy6qvOfVvXr18Pf39/aDQave3Dhg3D1KlTce7cOQwbNgw+Pj5wcnJCjx49sHv37pp/KJWcOHECDzzwABwcHNCiRQs89dRTuHLliu71pKQk9OzZE46OjnBzc0Pv3r3xzz//AACOHTuGvn37wtnZGS4uLujWrRt+bWB/ATeIIGT16tUICgqCvb09IiIicOjQIaNlExIS0L17d7i5ucHR0RHh4eH49NNP9cpMnjwZCoVC7zFo0KC6PgxZuEyXiEhyVaOBU3Ky2YfrL7/gQlmZ0XoEgMzSUrj+8ous+q5WCihMeeSRR5Cfn489e/botl26dAk7d+7Eo48+iitXrmDIkCFITEzE77//jkGDBmHo0KE4f/58bT4aAEBJSQkGDhwId3d3HD58GF9++SV2796N5557DgBQXl6O4cOH4/7778fx48eRkpKCp556CgqFAgDw6KOPIiAgAIcPH8aRI0cwb948NGvWrNbtsiSrzwnZunUrYmJisG7dOkRERCAuLg4DBw7EmTNnDKar9fDwwH/+8x907NgRdnZ2+PbbbzFlyhR4e3tj4MCBunKDBg3Cxx9/rHuuUqnq5XjMYU8IEVHj4e7ujsGDB2Pz5s3o168fACA+Ph6enp7o27cvbGxsEBYWpiu/ZMkSbN++Hf/73/90wUJNbd68GdevX8cnn3wCR0dHAMCqVaswdOhQvPXWW2jWrBmKiorw0EMPoW3btgCATp066d5//vx5zJkzBx07dgQAtGvXrlbtqQtWD0KWL1+OJ598ElOmTAEArFu3Djt27MBHH32EefPmVSnfp08fveczZ87Exo0b8csvv+gFISqVCr7aK74ZpaWlKK0wzlhcXFyDI5GHPSFERJLmNja4EhlpttzPhYUYcuKE2XLfdemC+9zcZO23Oh599FE8+eSTWLNmDVQqFTZt2oSxY8fCxsYGV65cwauvvoodO3YgKysL5eXluHbtmkV6Qk6dOoWwsDBdAAIAvXv3hkajwZkzZ3Dfffdh8uTJGDhwIPr374+oqCiMHj0afn5+AICYmBg88cQT+PTTTxEVFYVHHnlEF6w0FFYdjikrK8ORI0cQFRWl22ZjY4OoqCikpKSYfb8QAomJibqTUVFSUhK8vb3RoUMHPPvss8jPzzdaT2xsLFxdXXWPwMDAmh+UGewJISKSKBQKOCqVZh8DPDwQoFJBYaweAIEqFQZ4eMiqTztcIdfQoUMhhMCOHTuQkZGB5ORkPProowCA2bNnY/v27XjjjTeQnJyMo0ePokuXLigzMXxkSR9//DFSUlJwzz33YOvWrWjfvj0OHDgAAHj11Vdx8uRJPPjgg/jpp5/QuXNnbN++vV7aJZdVg5CLFy9CrVbDp9JaVR8fH2SbuEoXFRXByckJdnZ2ePDBB/Hee++hf//+utcHDRqETz75BImJiXjrrbewd+9eDB48GGq12mB98+fPR1FRke6RkZFhmQM0gD0hRETVo1QosCIkBACqBCLa53EhIVBWM7iQy97eHtHR0di0aRO2bNmCDh064K677gIA7Nu3D5MnT8aIESPQpUsX+Pr6Ij093SL77dSpE44dO4aSkhLdtn379sHGxgYdOnTQbbvzzjsxf/587N+/H3fccQc2b96se619+/Z44YUX8OOPPyI6OlpvmkJDYPXhmJpwdnbG0aNHceXKFSQmJiImJgZt2rTRDdWMHTtWV7ZLly7o2rUr2rZti6SkJN2YXkUqlare5ox07gy89x7QqlW97I6I6LYQ7eWF+NBQg3lC4uohT8ijjz6Khx56CCdPnsSECRN029u1a4eEhAQMHToUCoUCCxYsqLKSpjb7XLRoESZNmoRXX30VeXl5mDFjBh577DH4+PggLS0N69evx8MPPwx/f3+cOXMGZ8+excSJE3Ht2jXMmTMHo0aNQnBwMDIzM3H48GGMHDnSIm2zFKsGIZ6enlAqlcip1C2Qk5Njcj6HjY0NQm5GxeHh4Th16hRiY2OrzBfRatOmDTw9PZGammowCKlP3t5ALecqERE1SdFeXhjm6YnkwkJklZXBz84OkW5uddYDUtEDDzwADw8PnDlzBuPHj9dtX758OaZOnYp77rkHnp6eeOmllyw2r7B58+b44YcfMHPmTPTo0QPNmzfHyJEjsXz5ct3rp0+fxsaNG5Gfnw8/Pz9Mnz4dTz/9NMrLy5Gfn4+JEyciJycHnp6eiI6OxuLFiy3SNkuxahBiZ2eHbt26ITExEcOHDwcAaDQaJCYmVmtWsUaj0ZtYWllmZqbuBBERUeOlVCjQx9293vdrY2ODf//9t8r2oKAg/PTTT3rbpk+frve8OsMzlXOYdOnSpUr9Wj4+PkbneNjZ2WHLli2y92stVh+OiYmJwaRJk9C9e3f07NkTcXFxKCkp0a2WmThxIlq2bInY2FgA0iTS7t27o23btigtLcV3332HTz/9FGvXrgUAXLlyBYsXL8bIkSPh6+uLc+fOYe7cuQgJCdFbPWNNv/56K3W7zAU8REREtx2rByFjxoxBXl4eFi5ciOzsbISHh2Pnzp26yarnz5+HTYXlVCUlJZg2bRoyMzPh4OCAjh074rPPPsOYMWMAAEqlEsePH8fGjRtRWFgIf39/DBgwAEuWLGkwuUKmTQMOHwa+/hp4+GFrt4aIiOrDpk2b8PTTTxt8rXXr1jh58mQ9t8j6FKI6+WubiOLiYri6uqKoqAguLi4Wr3/oUODbb4H164Enn7R49UREDc7169eRlpaG4OBg2NvbW7s5VnH58uUqcyC1mjVrhtatW9dzi2rO1PmszjXU6j0hTRGX6RIRNT3Ozs5wdna2djMalAZx75imhgnLiKipYuf77cFS55FBiBWwJ4SImhrtjdOuXr1q5ZaQJWjPY21viMfhGCtgTwgRNTVKpRJubm7Izc0FIOW4qG76dLI+IQSuXr2K3NxcuLm5QalU1qo+BiFWwJ4QImqKtEkotYEINV5ubm6ybxJrCoMQK+jUCVi1CqjD++QRETU4CoUCfn5+8Pb2xo0bN6zdHKqhZs2a1boHRItBiBV4eQGVEuoRETUZSqXSYhcxatw4MZWIiIisgkGIlfz6KxAfz8mpRETUdDEIsZLp04FHHgEOHrR2S4iIiKyDQYiVcIUMERE1dQxCrIS5QoiIqKljEGIl7AkhIqKmjkGIlTAIISKipo5BiJVwOIaIiJo6BiFWwp4QIiJq6pgx1Uo6dgRWr2bqdiIiaroYhFiJlxcwbZq1W0FERGQ9HI4hIiIiq2AQYkXa1O1ZWdZuCRERUf1jEGJFM2ZIqdsPHLB2S4iIiOofgxAr4goZIiJqyhiEWBFzhRARUVPGIMSK2BNCRERNGYMQK2JPCBERNWUMQqyIPSFERNSUMQixIgYhRETUlDFjqhV16MDU7URE1HQxCLEiT0+mbicioqarQQzHrF69GkFBQbC3t0dERAQOHTpktGxCQgK6d+8ONzc3ODo6Ijw8HJ9++qleGSEEFi5cCD8/Pzg4OCAqKgpnz56t68MgIiKiarB6ELJ161bExMRg0aJF+O233xAWFoaBAwciNzfXYHkPDw/85z//QUpKCo4fP44pU6ZgypQp+OGHH3Rlli5dipUrV2LdunU4ePAgHB0dMXDgQFy/fr2+Dku2w4eBL78E/v3X2i0hIiKqXwohhLBmAyIiItCjRw+sWrUKAKDRaBAYGIgZM2Zg3rx5suq466678OCDD2LJkiUQQsDf3x8vvvgiZs+eDQAoKiqCj48PNmzYgLFjx5qtr7i4GK6urigqKoKLi0vND06G3r2B/fule8iMHFmnuyIiIqpz1bmGWrUnpKysDEeOHEFUVJRum42NDaKiopCSkmL2/UIIJCYm4syZM7jvvvsAAGlpacjOztar09XVFREREUbrLC0tRXFxsd6jvnCFDBERNVVWDUIuXrwItVoNH+2V+CYfHx9km8jgVVRUBCcnJ9jZ2eHBBx/Ee++9h/79+wOA7n3VqTM2Nhaurq66R2A9LldhwjIiImqqrD4npCacnZ1x9OhRHD58GP/9738RExODpKSkGtc3f/58FBUV6R4ZGRmWa6wZ7AkhIqKmyqpLdD09PaFUKpFT6Qqck5MDX20XgQE2NjYICQkBAISHh+PUqVOIjY1Fnz59dO/LycmBn5+fXp3h4eEG61OpVFCpVLU8mpphTwgRETVVVu0JsbOzQ7du3ZCYmKjbptFokJiYiF69esmuR6PRoLS0FAAQHBwMX19fvTqLi4tx8ODBatVZX9gTQkRETZXVk5XFxMRg0qRJ6N69O3r27Im4uDiUlJRgypQpAICJEyeiZcuWiI2NBSDN3+jevTvatm2L0tJSfPfdd/j000+xdu1aAIBCocCsWbPw+uuvo127dggODsaCBQvg7++P4cOHW+swjWJPCBERNVVWD0LGjBmDvLw8LFy4ENnZ2QgPD8fOnTt1E0vPnz8PG5tbHTYlJSWYNm0aMjMz4eDggI4dO+Kzzz7DmDFjdGXmzp2LkpISPPXUUygsLMS9996LnTt3wt7evt6Pz5z27YE1a4CAAGu3hIiIqH5ZPU9IQ1SfeUKIiIhuJ40mTwgRERE1XQxCGoBDh4CtW4ELF6zdEiIiovrDIKQBiIkBxo6V0rcTERE1FQxCGgAu0yUioqaIQUgDwGW6RETUFDEIaQDYE0JERE0Rg5AGgD0hRETUFDEIaQDYE0JERE0Rg5AGgD0hRETUFFk9bTsB7doBa9cCLVtauyVERET1h0FIA+DhATzzjLVbQUREVL84HENERERWwSCkgdCmbs/MtHZLiIiI6geDkAZi9mymbicioqaFQUgDoV0hw2W6RETUVDAIaSC0uUK4TJeIiJoKBiENBBOWERFRU8MgpIFgwjIiImpqGIQ0EOwJISKipoZBSAPBnhAiImpqmDG1gQgJAdatY+p2IiJqOhiENBDu7sDTT1u7FURERPWHwzFERERkFQxCGpBDh4DPPwcyMqzdEiIiorrHIKQBmTsXGDcO2LfP2i0hIiKqewxCGhCukCEioqaEQUgDwlwhRETUlDAIaUC8vaWvKSlAUhKgVlu1OURERHWKQUgDkZAALFsmfb93L9C3LxAUJG0nIiK6HTEIqQdqIZBUUIAtOTlIKiiAWgi91xMSgFGjgIIC/fdduCBtZyBCRES3owYRhKxevRpBQUGwt7dHREQEDh06ZLTsBx98gMjISLi7u8Pd3R1RUVFVyk+ePBkKhULvMWjQoLo+DIMS8vIQdOAA+h47hvGnTqHvsWMIOnAACXl5AKQhl5kzgUpxCYBb22bN4tAMERHdfqwehGzduhUxMTFYtGgRfvvtN4SFhWHgwIHIzc01WD4pKQnjxo3Dnj17kJKSgsDAQAwYMAAXLlzQKzdo0CBkZWXpHlu2bKmPw9GTkJeHUSdPIrO0VG/7hdJSjDp5Egl5eUhOBjIzjdchhJQ3JDm5jhtLRERUz6wehCxfvhxPPvkkpkyZgs6dO2PdunVo3rw5PvroI4PlN23ahGnTpiE8PBwdO3bEhx9+CI1Gg8TERL1yKpUKvr6+uoe7u7vRNpSWlqK4uFjvUVtqITAzNRUGOjh022alpuJClqESVWVl1bpJREREDYpVg5CysjIcOXIEUVFRum02NjaIiopCSkqKrDquXr2KGzduwMPDQ297UlISvL290aFDBzz77LPIz883WkdsbCxcXV11j8DAwJodUAXJhYVVekAqEgAySkuR51coqz4/v1o3iYiIqEGxahBy8eJFqNVq+GgTZNzk4+ODbJkZu1566SX4+/vrBTKDBg3CJ598gsTERLz11lvYu3cvBg8eDLWRiRXz589HUVGR7pFhgbzpWWVlssp5dShDQACgUBh+XaEAAgOByMhaN4mIiKhBadR30X3zzTfx+eefIykpCfb29rrtY8eO1X3fpUsXdO3aFW3btkVSUhL69etXpR6VSgWVSmXRtvnZ2ckq19LeDitWSKtgFArDE1Tj4gCl0qLNIyIisjqr9oR4enpCqVQip1KK0JycHPhqc5gb8c477+DNN9/Ejz/+iK5du5os26ZNG3h6eiI1NbXWbZYr0s0NASoVjHRwQAEgUKVCpJsboqOB+HigZUv9MnZ20vbo6LpuLRERUf2zahBiZ2eHbt266U0q1U4y7dWrl9H3LV26FEuWLMHOnTvRvXt3s/vJzMxEfn4+/OpxYoVSocCKkBAAMBqIxIWEQHlzHCY6GkhPB/bsAVavlnpFysqALl3qp71ERET1zeqrY2JiYvDBBx9g48aNOHXqFJ599lmUlJRgypQpAICJEydi/vz5uvJvvfUWFixYgI8++ghBQUHIzs5GdnY2rly5AgC4cuUK5syZgwMHDiA9PR2JiYkYNmwYQkJCMHDgwHo9tmgvL8SHhqJlpaEeRxsbxIeGItrLS2+7Ugn06QNMmwZo05ps3FhPjSUiIqpnVg9CxowZg3feeQcLFy5EeHg4jh49ip07d+omq54/fx5ZFdanrl27FmVlZRg1ahT8/Px0j3feeQcAoFQqcfz4cTz88MNo3749Hn/8cXTr1g3JyckWn/chR7SXF9Lvvht7wsIw9+aqGw9bW4zw9DT5vsmTpa8bNzJRGRER3Z4UQhiaCtm0FRcXw9XVFUVFRXBxcbFYvSVqNTx++QVlQuB0z57o0Ly50bLXrwP+/lIq9x9/BPr3t1gziIiI6kx1rqFW7wlpShyVSkS6ugIAfrh0yWRZe3tg3DggIsL48l0iIqLGjEFIPRtwM6naj2aCEEBamnvgAFAhBQoREdFtg0FIPRtwM338nsJClGo0Jss2a1YfLSIiIrIOBiH1rKuTE3yaNcNVjQYpRUWy3lNQAHz5ZR03jIiIqJ4xCKlnNgoF+t8ckvmhoMBs+cuXpbTto0cDp0/XdeuIiIjqD4MQK9AOyciZF+LsDPTtK33PnCFERHQ7YRBiBf1vBiG/XbmCPBk3uruZtw2ffMKcIUREdPtgEGIFvioVwhwdAQC7ZAzJPPQQ0KIF8O+/wK5ddd06IiKi+sEgxEqqs1TXzg4YP176fsOGOmwUERFRPWIQYiUDtUFIQQHkJK3VDsl89ZW0WoaIiKixYxBiJb1dXOBgY4OssjL8UVJitnx4ONC1K1BeDuzbV/ftIyIiqmsMQqzEXqnE/W5uAKTeEHMUCuCjj4DMTGmOCBERUWPHIMSKBlZjqS4AdOsG+PpKK2SSkoAtW6SvXDFDRESNEYMQK9JOTv25qAjXZEYSCQlAUJCUO2T8eOlrUJC0nYiIqDFhEGJFnZo3R0s7O1zXaJAsI4V7QgIwapQ0JFPRhQvSdgYiRETUmNQoCNm4cSN27Nihez537ly4ubnhnnvuwT///GOxxt3uFArFrVUyZoZk1Gpg5kzA0EIa7bZZszg0Q0REjUeNgpA33ngDDg4OAICUlBSsXr0aS5cuhaenJ1544QWLNvB2N0DmfWSSk6v2gFQkBJCRIZUjIiJqDGxr8qaMjAyEhIQAAL766iuMHDkSTz31FHr37o0+ffpYsn23vSh3dygA/FFSgn9LS+GvUhksl5Ulrz655YiIiKytRj0hTk5OyM/PBwD8+OOP6N+/PwDA3t4e165ds1zrmoAWzZqhu7MzANMp3P385NUntxwREZG11SgI6d+/P5544gk88cQT+OuvvzBkyBAAwMmTJxEUFGTJ9jUJ2rvq/mBiXkhkJBAQIOULMUShAAIDpXJERESNQY2CkNWrV6NXr17Iy8vDtm3b0KJFCwDAkSNHMG7cOIs2sCnQzgvZVVAAjZEU7kolsGKF9H3lQET7PC5OKkdERNQYKIScG5c0McXFxXB1dUVRURFcXFzqfH83NBp47NuHK2o1jnTrhrtuDs8YkpAgrZKpOEk1MFAKQKKj67ypREREJlXnGlqjnpCdO3fil19+0T1fvXo1wsPDMX78eBTw7mrV1szGBg/cTOFuakgGkAKN9HRgzx5g82bpa1oaAxAiImp8ahSEzJkzB8XFxQCAEydO4MUXX8SQIUOQlpaGmJgYizawqdAOyWzNzcWWnBwkFRRAbWJopk8fYNw46SuHYIiIqDGq0RLdtLQ0dO7cGQCwbds2PPTQQ3jjjTfw22+/6SapUjXdDDiOlZRg/KlTAIAAlQorQkIQ7eVl9C1//QUcPgxMmFBvLSUiIrKIGvWE2NnZ4erVqwCA3bt3Y8CAAQAADw8PXQ8JyZeQl4cZqalVtl8oLcWokyeRkJdn8H2XLgGdOgGPPQZkZ9d1K4mIiCyrRkHIvffei5iYGCxZsgSHDh3Cgw8+CAD466+/EBAQYNEG3u7UQmBmaioMDbxot81KTTU4NNOiBRAeLn2fmFhXLSQiIqobNQpCVq1aBVtbW8THx2Pt2rVo2bIlAOD777/HoEGDLNrA211yYSEyS0uNvi4AZJSWIrmw0ODrUVHSVwYhRETU2NRoTkirVq3w7bffVtn+7rvv1rpBTU1WWVmtyvXrB7z9NrB7tzRHxFgyMyIiooamRj0hAKBWq7Ft2za8/vrreP3117F9+3aoa3gL19WrVyMoKAj29vaIiIjAoUOHjJb94IMPEBkZCXd3d7i7uyMqKqpKeSEEFi5cCD8/Pzg4OCAqKgpnz56tUdvqmp+dXa3K3XsvYGcn3bzOwLQSIiKiBqtGQUhqaio6deqEiRMnIiEhAQkJCZgwYQJCQ0Nx7ty5atW1detWxMTEYNGiRfjtt98QFhaGgQMHIjc312D5pKQkjBs3Dnv27EFKSgoCAwMxYMAAXLhwQVdm6dKlWLlyJdatW4eDBw/C0dERAwcOxPXr12tyuHUq0s0NASoVjHVgKAAEqlSIvJlHpDJHR6BXL+n73bvrooVERER1o0YZU4cMGQIhBDZt2gSPm/kt8vPzMWHCBNjY2GDHjh2y64qIiECPHj2watUqAIBGo0FgYCBmzJiBefPmmX2/Wq2Gu7s7Vq1ahYkTJ0IIAX9/f7z44ouYPXs2AKCoqAg+Pj7YsGEDxo4da7bO+s6YmpCXh1EnTwKA3gRVbWASHxpqdJkuALz+OrBgATBqFPDll3XXTiIiInPqPGPq3r17sXTpUl0AAgAtWrTAm2++ib1798qup6ysDEeOHEGUdnYlABsbG0RFRSElJUVWHVevXsWNGzd0bUlLS0N2drZena6uroiIiDBaZ2lpKYqLi/Ue9SnaywvxoaFoqVLpbXdRKs0GIAAwfjzw7bfARx/VZSuJiIgsq0ZBiEqlwuXLl6tsv3LlCuxkznEAgIsXL0KtVsPHx0dvu4+PD7JlJr546aWX4O/vrws6tO+rTp2xsbFwdXXVPQIDA2Ufg6VEe3kh/e67sScsDI/7+gIAQhwczAYgANCmDfDgg4CJW84QERE1ODUKQh566CE89dRTOHjwIIQQEELgwIEDeOaZZ/Dwww9buo1Gvfnmm/j888+xfft22Nvb17ie+fPno6ioSPfIyMiwYCvlUyoU6OPujjfatIENgCNXriDt2jWrtIWIiKiu1SgIWblyJdq2bYtevXrB3t4e9vb2uOeeexASEoK4uDjZ9Xh6ekKpVCInJ0dve05ODnxv9gYY88477+DNN9/Ejz/+iK5du+q2a99XnTpVKhVcXFz0HtbkbWeH+29ORN1mJFtqZefPAy+/LN1hl4iIqDGoURDi5uaGr7/+Gn/99Rfi4+MRHx+Pv/76C9u3b4ebkVUchtjZ2aFbt25IrJBpS6PRIDExEb20Sz4MWLp0KZYsWYKdO3eie/fueq8FBwfD19dXr87i4mIcPHjQZJ0NzaibwzBfygxCLl8GYmOB9euBBrgIiIiIqArZycrM3R13z549uu+XL18uuwExMTGYNGkSunfvjp49eyIuLg4lJSWYMmUKAGDixIlo2bIlYmNjAQBvvfUWFi5ciM2bNyMoKEg3z8PJyQlOTk5QKBSYNWsWXn/9dbRr1w7BwcFYsGAB/P39MXz4cNntsrZoT088d/YsDl2+jH+uX0drM8NNnTsDvr7SPWT27wceeKCeGkpERFRDsoOQ33//XVY5RTVTdo4ZMwZ5eXlYuHAhsrOzER4ejp07d+omlp4/fx42Nrc6bNauXYuysjKMGjVKr55Fixbh1VdfBQDMnTsXJSUleOqpp1BYWIh7770XO3furNW8kfrmq1Ih0tUVPxcVYVteHmLMTJZVKKTsqZs2SSncGYQQEVFDV6M8Ibe7+s4TYsyqzEzMSE1FLxcX7L/rLrPlN2wApkwBIiKAAwfqvn1ERESV1XmeEKof0V5eUABIKS5GpoyJHv36SV8PHwaM3O+OiIiowWAQ0oD5q1To7eoKANh28aLZ8oGBQPv2gEYDVCNnHBERkVUwCGngtKtk4mWukunXD2jRAsjPr8tWERER1R6DkAZupKcnAGBfURH+LS01W/6tt4DcXGDq1LpuGRERUe0wCGngAuzt0cvFBQLSje7McXYGbHhWiYioEeDlqhGobuIyABACKCmpqxYRERHVHoOQRkAbhCQXFSFbxpDMF18AAQHAtGl13TIiIqKaYxDSCLSyt0dPZ2cIANtlrJLx8AD+/RfYvVvqESEiImqIGIQ0EtUZkundG1CppEDkzJm6bhkREVHNMAhpJLRByN7CQuSWlZks6+AgBSIAsHYtsGULkJQEqNV13EgiIqJqYBDSSAQ7OKCbkxM0kDck4+srfV25Ehg/HujbFwgKAhIS6rSZREREsjEIaUQe8fYGYD5xWUICsHlz1e0XLgCjRjEQISKihoFBSCOiHZLZU1CAi0aGZNRqYOZMw+/XTlKdNYtDM0REZH0MQhqRtg4OuNPJCWoAb50/jy05OUgqKIC6whKY5GQgM9N4HUIAGRlSOSIiImuytXYDqHo6NW+O369cwTsVIo0AlQorQkIQ7eWFrCx59cgtR0REVFfYE9KIJOTlYUtubpXtF0pLMerkSSTk5cHPT15dcssRERHVFQYhjYRaCMxMTYWh3GPabbNSU3HPvQIBAYBCYbgehQIIDAQiI+uqpURERPIwCGkkkgsLkWkiZbsAkFFaiv2XC7FihbTNWCASFwcolRZvIhERUbUwCGkksswkKKtYLjoaiI8HWrbUf02lkrZHR9dBA4mIiKqJQUgj4WdnV61y0dFAejqwZw+wYgVgYwOUlkpDMURERA0Bg5BGItLNDQEqFYyMsAAAlABsKo7B2AggrABeY3LQf04BYCPwxht13VIiIiJ5uES3kVAqFFgREoJRJ09CARicoKoG0PfoUcxt1QrhTk6Yfe7crXkkgwDcpcJXq0Jw8qQXQkPrr+1ERESGsCekEYn28kJ8aChaqlR62wNVKnzSsSMm+fhAA+DN8+cx9s8/q05k9SoFFp/EtC3m78RLRERU1xRCCEN/VDdpxcXFcHV1RVFREVxcXKzdnCrUQiC5sBBZZWXws7NDpJsblDeHYeJzczHmzz+hMfZmAXioVcjtd7fuPURERJZSnWsoh2MaIaVCgT7u7gZf82zWzHgAAgAK4JJtKZILC43WQUREVB84HHObqc5SXiIiImtiEHKbkbuU98B3dnj77TpuDBERkQkMQm4z5pbyKgB4CRVWPuGGxYuBS5fqs3VERES3MAi5zWiX8gIwGoisDQ1BeFcFSkqA996rv7YRERFVZPUgZPXq1QgKCoK9vT0iIiJw6NAho2VPnjyJkSNHIigoCAqFAnFxcVXKvPrqq1AoFHqPjh071uERNDzGlvIqAHzcsSNGenvh5ZelbStWAJcv138biYiIrBqEbN26FTExMVi0aBF+++03hIWFYeDAgcg1cLt6ALh69SratGmDN998E76+vkbrDQ0NRVZWlu7xyy+/1NUhNFjRXl5Iv/tu7AkLw6ZOndDW3h4CQNq1a9Lr0UCHDkBBATB3LrBlC5CUBKjVVm02ERE1IVYNQpYvX44nn3wSU6ZMQefOnbFu3To0b94cH330kcHyPXr0wNtvv42xY8dCVemv/IpsbW3h6+ure3h6etbVITRo2qW84318ENumDQBg5YULuFxeDqUSiIqSyq1bB4wfD/TtCwQFAQkJ1mszERE1HVYLQsrKynDkyBFEaa+EAGxsbBAVFYWUlJRa1X327Fn4+/ujTZs2ePTRR3H+/HmT5UtLS1FcXKz3uN1Ee3mhg4MDCsrLsfbff5GQAKxZU7XchQvAqFEMRIiIqO5ZLQi5ePEi1Go1fHx89Lb7+PggOzu7xvVGRERgw4YN2LlzJ9auXYu0tDRERkbisomJD7GxsXB1ddU9Am/DW80qFQrMb90aALA8IwPPz1bDUK5c7bZZszg0Q0REdcvqE1MtbfDgwXjkkUfQtWtXDBw4EN999x0KCwvxxRdfGH3P/PnzUVRUpHtkZGTUY4vrz3hvbwTZ2yPnxg1cCMsyWk4IICMDSE6ux8YREVGTY7UgxNPTE0qlEjk5OXrbc3JyTE46rS43Nze0b98eqampRsuoVCq4uLjoPW5HzWxsMFfbyzM2A7A1meAdWcbjFCIiolqzWhBiZ2eHbt26ITExUbdNo9EgMTERvXr1sth+rly5gnPnzsHPz89idTZmU3x90QJ2gE8pMCDHZNmafGRqtbTKhqttiIjIHKsOx8TExOCDDz7Axo0bcerUKTz77LMoKSnBlClTAAATJ07E/PnzdeXLyspw9OhRHD16FGVlZbhw4QKOHj2q18sxe/Zs7N27F+np6di/fz9GjBgBpVKJcePG1fvxNUT2SiVeCr7ZGzLuPGBjvDdk795bQYSc4CIhQVpd07cvV9sQEZEMwsree+890apVK2FnZyd69uwpDhw4oHvt/vvvF5MmTdI9T0tLEwCqPO6//35dmTFjxgg/Pz9hZ2cnWrZsKcaMGSNSU1Or1aaioiIBQBQVFdX28BqkyzduCKfEZIE9ewT6ZQtpFoj0UCiE3vN+/YT48EMhAgL0twcECLFt2606t22r+l5tfQqFflkiIrp9VecaqhDC0BqJpq24uBiurq4oKiq6beeHLElPx8L0dNhmOKJ8UndASEneAwOBuDgpi+q0acDVq4bfr7iZEz4+Hhg2TOrxyMw0XjYgAEhLA5RKix8KERE1INW5htrWU5uogXmuZUu8nZGBy4ElmPrTP2he4IC27naYdq8b7GylCKN7d+DOO4EbN6q+XwgpuJg2Dfj9d+MBiLasdrVNnz51czxERNT4MAhpotybNUOUuzu2X7yIj5AOuEvblx1WYUVICKK9vJCXZzgA0RICyMkBXn9d3j7rcrWNWi0FOVlZ0oTayEj2uhARNXS3XZ4QkichLw9fXbxYZfuF0lKMOnkSCXl5+kGDjQDCCoAHcqSvNrdG8eTmdqurBUqcEEtE1DixJ6QJUguBmampMDQZSEC62+6s1FR87OcpPYvMA55LBbxLbxXMVQGrQoBkL3z8MTB5spTy3dgMo4AAqXfC0hISpDTzlferTT8fHy/drI+IiBoe9oQ0QcmFhcgsLTX6ugCQUVqK/I558BiRCyw+CXhVKu9ZCiw+iRYj8tCnD7BihbRZO2G1suBg46/VlFoNzJxpOPBh+nkiooaPQUgTlFVWJqvcmNN/4tKMP6UnlQMIG0jRyvRUwEYgOlrqdWjZUr+YpydgYyPN15g923hPiTGm8pMkJ8ufEEtERA0Pg5AmyM/OTlY5hfYfYz0YNkC+shTJhYUApGGP9HRgzx5g82bpa3Y2sGGDVPzdd4G335bfTlNzPYqKgO++k1cP088TETVMnBPSBEW6uSFApcKF0lKD80IUAAJUKrweFIRJZ86Yra9iz4pSWXUZ7mOPAbm5Uk/Ili3SEIqtrenVLMbmemRmAiNHSu8vL5d3vMzYT0TUMLEnpAlSKhRYERICoGonh/Z5XEgIWtnby6pPzn+iF18E1q+XhlR27DC9msXUXA+t8nKgfXvA29v4XBOFQlq5UxcTYomIqPbYE9JERXt5IT40FDNTU/UmqQaoVIi7mSdELYTJHhOtp86cARQKjPH2hloIJBcWIqusDH52doh0c4PyZpTw5JPmV7O89ZYUOJia66G1bh1QUCC9T6EwHLTExTFfCBFRQ8W07QY0hbTtWqaCBkDKJzLq5EkA0AtEtCVC7O1x9vp1AEAfV1f8de0a/q0wPBOgupX8TK02nd5da/hw4KuvzLd982Zg3DgpsJk5U79epVJ6ffRo8/UQEZHlVOcayuGYJk6pUKCPuzvG+figj7u7XgAC3OoxaalS6W0PUKkQHxqKkz174pXWraEAkFRUpBeAAPrJz/RWs5hIfnb5sry2a+d6VJwQu2ED4OEhDelcuiT/cyAiovrH4RgyK9rLC8M8PY32mLwaFIT3//0XeQZyvFdMfhabJS/52ZQpwJkzxpOfaW+IV3GuR8UJsUVF0hyUCxcs9QkQEVFdYBBCsmh7TAxJLiw0GIBoaZOfnfHLBSJtpORnld1MfoZFoWjZ0gsrVhie66HtqDE11+Opp4CHHgLatJF3bEREZB0MQqjW5CY/W4JTwKKbTwwlP9MAypmpuOdeT9jZKhAfDzw/S+CCRyHQogzIt0PLAjeseFdhMhW7vT0DECKixoBBCNVadZKfCVMrVWwAdYtS7L9cKPW6ROZB8XkqUFZh2MZOBbQLAeAla58nTkjDMoMGySpORET1iBNTqda0yc+MJVZVAAhUqbC+fXtZ9b2bmYlFaWkYdfIkMsv071lzoezWRFdzfvgB6NoVePxx4OYCHllMpYqvCbUQSCoowJacHCQVFEDNBWlERADYE0IWoE1+NurkSam3o8JrFZOfedjK++/2v/x8/C8/3+BrFSe6DvP0rLKap6I+faScIxkZUqK06TNML0cGDC/3DQiQbtBXk7vxJuTlGczFol22TETUlLEnhCzC3FLeaC8vWT0mHra2GGBkAqyWdqKr9p41xqhUwH/+I32/aHceWqccQN9jxzD+1Cn0PXYMQQcO6PWoaBOpVc5jok2kps3oKpc2x0rlOxZXXLZcEXtMiKipYbIyA5pSsjJLq23ys/jQUJRqNBh/6pTZfW3u1AnjfHxM7resDAgYl4e8507q76jSPod5eN1KpGYjgC6FusmwOOEGhVAgIABIS5OXgVUtBIIOHKgSgFTcd4BKhbS774ZSoWCPCRHdNqpzDeVwDFmUqaW8gLx08UkFBbL29fXFixjg4YG9hYVGL+DDPD1R/kyqtLFSF0zFoR3XUk9kZhrPYSJWhSAj2QvJybfykZgKuJILC40GINp9a3tzLpWXY9TJk1VS42t7TLQ9SUREtxsGIVTvzCU/M3eXX62teXn4X34+rmk0VV7TXsAXtW6Ngmbmg4HdOYVAZLnZHCZZWVIwYK7n4uTVq3I+CryYmorU69cNHmd15r8QETVGHI4xgMMx1mdu2GZ+q1b49uJFHJd5sTfH9g8XlLe+CjiVV81hAgAaAHkq7Pa5G0WhFw32XGgn5fZ0dsavly+jamhUc3vCwnQ9TGXlAmt+KcS5gjK0dbfDtHvdYGfLAIWIGgbeO4YaPXMTXf/bpg3ebdfOYvsrv6MYcDYSgADST4pPKVJD/sVzZ88a7bkAgEM3AxA7Ez0XCgDezZrhUW9vWe3TJoSb+00emn99AC/gGFa5n8ILOIbmXx/A3G/ML1kmImpoOBxDDZa5YZscmZlaFVdsIZqXGw65NQCKm6EznPGnm/k73j2TelbWPj9q3x6uzZqZ7M1Z2749PGxtsSk312x93+TnY8+v5fjAqer+1e6leFtxEvgmFEuHcu4IETUe7AmhBs3UXX7lZmoVX7SUrvyVx0c0kLYvb48n3QJl1dVc5rwMe6XSIsuWtbbk5uKD5jcDEEMp7wWwvCwVZeVSuMPlvkTUGLAnhBotcxNYFQA81CrkbwoC0p2qrnrJUwGrpTv3emUJBASZmAyrAZoVqfC/+zsi6vgxs23TBkjmenPkJHqbExiIrzIK8JfiivEd3kx5v+aXQrQKLedyXyIyy1xKhfrAIIQaLTkX8OeVIVikUQDJXsA+zyr5P6CRSrb0M12XUAAr2oWgj7v5wCdApUKkm5teO80tW559ORTLy1KhbnErcLDJVyHGLgRzXL3w0/ocYLD53CnxV7Ow/2Qul/sa0BB+4RI1FA0lNxFXxxjA1TGNi6EfpsCbeUe0ScguXAAM/U9XKKCXhMxUXdofTDkJ16rzQ6zN1CoUlZKk/eEGqBWwswPKOhUAceZ7YEypnCANsPyFuaFe6BvKL1yihkD7O8zQCj+g+r/DKqvONZRBiAEMQhofUxc/7UUe0A9EtNfG+Hj9+8KYu5CePQuccDMfrOjqUwPJyUBWFuDnB0RG3sq6qlbjVqZWE8LvEjg+/wA0HqWGZ3IJAOUAmpmuB7i13NfSF+aGeqGv61+4RI1JdbM510SjWqK7evVqBAUFwd7eHhERETh06JDRsidPnsTIkSMRFBQEhUKBuLi4WtdJtwdTE1ijo6VAo2VL/fcEBFQNQEzVJQTwxBNA+/aA+x9eSL/7buwJC8PmTp2wJywMaXffXeVilpAgBRl9+wLjx0tfg4Ju3YcmOdl8AAIAy99R4EVViPEJtgDuLwgwXxGA9VlZWHvhQrXua2NOde+TU1/UQmBmaqrJJdWzUlM5cZeajOpkc64PVg1Ctm7dipiYGCxatAi//fYbwsLCMHDgQOQaWbJ49epVtGnTBm+++SZ8fX0tUic1DdHRQHo6sGcPsHmz9DUtrXp3xlUoAO2CnBkzAE258cAHMH9DvOXLgXXr5O07OxtYOtQLc66EQlmgv9oGF1XAolD8taGFrLq25OZimplcJ5UvzKZW29TkQq9WA0lJwJYt0le1WlbTq62h/cIlqg+mfl7/vn5dVh1ZMlMg1JZVh2MiIiLQo0cPrFq1CgCg0WgQGBiIGTNmYN68eSbfGxQUhFmzZmHWrFkWq1OLwzFkzKVLUk9Ifr4URNx5Z+2GWeTas+fWPWsqZ0wd3dEN996jQNo/AqqEAyhzNT5p1s3WFq1VKhwtKTG/T5nDNltycmTdcFBXXwIwc6b+ZxMQAKxYUb2gUMvU8JnctlW8GSJRY2bs5/X14GCkXbuGZRkZuGLgVheVVczSXF2N4gZ2ZWVlOHLkCObPn6/bZmNjg6ioKKSkpNRrnaWlpSitcMKKi4trtH+6/Xl4AP/9L/DMM8CLL+rPMal4IZU7zBIRAZw5AxQVmZ44Gxl5a5udrQKz+uj/cvjf/4BevRS4siwEWHwSCoXhSbMfdugg+y7FsefPY0d+PpZlZhpcbTPy5Em0Vqnwj4mehoqyyspuTcKtVKG2d8jQ8JgppgKkAe7u+FlmD4fcnDNUfxrqJOeGzNj8p8zSUkw+fVr33BbSFDJDDK3wq0tWC0IuXrwItVoNn0p/ffj4+OB0hQ+rPuqMjY3F4sWLa7RPanpa3Bz1MHYh/fBDYP9+eXXNnAmoVNL7FArDE2fj4m71sBhzxx3AZ58BI0Z4QSwKheuCVBRWuHFfywo9F3LvUvxjQQF+NFJW20y5AQgAeNvaYfJMw8GWENLxzpoFDBtm/ngB479wtQGSk42NrL/43Gxtca+rq6xjoPphzUnOcoMfS5ezRLuNDYtqNVMo8EnHjrBVKDD6zz8BGP5jJS4kpN4CPuYJATB//nzExMTonhcXFyMwUF4GTWpa1GrghRcMv6a9uD7+uPz6/PykYZb4eMNDFHFx8nsGhg0DXn8d+M9/vHBtuCfQrlC33FdccgPiFEA0cI+zG5T5Kqjdjay00QA2JbZ4qJUr/leQb3a/2zp3xsxz50ze9dhOoUDxCUeTvUNCABkZUi+SdujJGDnzUK5oNGhrb49BHh5Y8++/eq9VVFhejulnz2JVu3ZoZmP1ufqNmiUuuKaCy7rOdSM3+LF0OUswN/8JAG4IAV87O/Rxd0e8QmGwbYZW+NUlqwUhnp6eUCqVyMnJ0duek5NjdNJpXdWpUqmgqpRam8gQucMsHTpIk0mLi+UNs0RHS0GEsaW8cnXoIH0tvaoAjt0asvlXcWu4w8FBAfUKadgGGugHIjdT2Wve7oCur2nwP5gPQkqFMJroTatMCDxXfhTw6QLkOAA2wmjiuKysW+8zdlGT8wsXAN7v0AH93N3xgLu7wSXVD7i54ZOcHKzPysLZa9fwZWgo3GxtG/0wgDX+SrfEBddccKmANMl5mKenxc+J3ODH0uUsRe5EUm05c9mc64vVghA7Ozt069YNiYmJGD58OABpEmliYiKee+65BlMnUUUVL5CmLFpU/WEWpdJ8D4AparU0nGGIdv+PPAJIIxRewKJQk6nskV8AyFhw46f9yyo01OCF/oWAACzPzEQmrgKrfwe2BgCjLujvN1cFrJL26+cnbTJ1USuRuZwmV8Yv3FFeXhh36hT2FBbijsOHASGQfeNGlX1Wyf/SQBO91cVf6ebaVqML7tGjwPz5QGwsEB4OoHqrmao7adLUMcjpWXv2r7/gbmuLZ/76y2yQ9FCLFvUeTHk2k5EkCPrzn8xlc64PVh2OiYmJwaRJk9C9e3f07NkTcXFxKCkpwZQpUwAAEydORMuWLREbGwtAmnj6581xrLKyMly4cAFHjx6Fk5MTQkJCZNVJVBvaC6SccpYaZpFLTi+N3hQJM6ns+7Rww4ZqpKg3daF/xMsLQ46fwAmUAM/+XbUyz1KpZ2ZRKE6f9kJ+5zw88qfx+R6OModN5PzCfcjTE/vvvBP9jh1DtoG/Jg1dSBtqore6+CvdXNtq3HuxbRuwcyfQo4cuCKnuX/NymTsGOT1ruTdu4IFjprMWa4Mkp+Rk3DCx8LQ2wZQhl27cwH//+cdkmfqecCqX1TOmrlq1Cm+//Tays7MRHh6OlStXIiIiAgDQp08fBAUFYcOGDQCA9PR0BAcHV6nj/vvvR1JSkqw65eASXTJGu/RWbhp47XtqO8wix5YtUkI0c95/H1iyxPgxAEBgoHQMX1+yXIr6SzduwCd5P8ptjOxUA6knZkIEmm8/iKuOpVXvGFyBDarmbavYvupkfVQLgcCUFKMXt4r1fX3xokUzsFoqo6ucTJgtVSqk3Hknuh85gpwKvT2Vy8k91jXt2uFCaSleP3/ebPuqLPkMDweOHZO+/v47ACCpoAB9zVzoDdZlgrnP9xEvL+wrKsIFGYGNi1KJYgsmtanu0nBDvTmp167hoRMnkHrtGuxtbHBdozF6L636yg7MtO21xCCETKluGvj6kpQkZWQ1Z88eKd+JoWPQevVVaUgJkHc/HVntk3mBQY4d4GP+grCwdWssufnXX21/4cpt26Pe3th56RLyyw0vcKxJ8GOpFNqyP1+Zejo54c+rV2WtMJJD74KbkwNUnKeXkwN4e6NUrYbHvn24amKfLkol8nv3hq2M3jBzn291vdu2LV44d85suZdbtcIbNQnMTDD0c+jZrBmuqtW4qtGgtUqFb7p0wdlr1yzy81objSptO1FjU9008PUlMlJqg7FrlUIh9XBERho/hubNpa8rVgB//SV9H+0lL0W9ObK70GUEIADQ3r454kND0bLSpPIAlaraf/HJbdum3FyjAQhgOAOrqeyV1c3oaqouS2e4PHTliqwAJFBmjhW9XCw//KD/4g8/4LpajTF//mkyAAGAYrUa08+elZVqX+4E5rkBAfC3szPa8aaAdCGf1rIlAlQqs+UWBQWZLAcADjY2CHFwAGD6vALGb4tw8cYNXNVo0MHBAYe6dUMXJyeL/bzWFy7RJaoBS61msSSlUgoe5E6GNXQMPXoAUVHAgQPA0KHSV3d3y0xgk5sQbAh88R2yzZbLO2OHWX3cLTLDX27bwhwdcUxGttl/bwYEpuYi9HR2xrqbS4fNySorM1lXb1dXfJNvfiUTALzTpg1m/21gXk4lQzw88N2lS2bLxbZpg3lpaSaXaCsBOFf44dDs2AEolbBRq6FRKlH+zTcY3KULkgoLoVIoMCsgAJtyc6v8NT/YwwMfZGVhfVYWLt64gU2dOqGZjY3R8y83MAt3dkaEq6vBFV4Vc2fY2dgYXQkmt5zWNY0GXX/9FRN8fLA9Lw+ZFdoqd86NVolGgxYVJqY2hAmncnE4xgAOx1BjZigtemCg/MmwOTlSMJKRAfzf/wFTp1qmXdqucXMTXf+bHoGJyoPSZFUjeUyQp8Jnmrvx6DjLrCzQddtfNzIPRQCB9ip83LEjomQMebRSqXCPiwu25uWZvHjIdUfz5vjj6tUq27UXOFMZMCuWDVCpkBoRgbYHD5o9D3KPdU9YGC6Vl+vNHfLPy4NPpUR3dgoF5rZqBVcbG/QcPhzOFY6nyNERDyxbhuZKJd4NCUF3FxeohcDvly/j4o0bcPH3R0TnzlAqFNiWl4fxf/6JMiHQuXlzFJaX64I+4NYF3LtZMzx39qysoNHULQoMDWXUttwLAQH4LCcHv125YrA92v+Cn3fujMvl5XhC2y0p4xgaAs4JqSUGIdTY1XYy7PHj0lzBSZMsU9/evYCnJ3DG2/xEV4+TXui7ME9aLSNgMI8JFoViz2tetVrSXNncb/LwtpPxfc65EorYhzxNBlLVda+zM05eu4bC8vJa1xfh7Iy+bm54KyMDgOl5MtrufVPlhnmaPtbK81UqXnB3x8Sg383JpoZoFArYVLj0VH5eRb9+wO7duqd7Cgow5MQJXK/lfBVDc27qK2PqdbUaPvv3W2yia0O6/xGDkFpiEEJ0S21vOKfRAJ06SXNMtm0DEGn6r0jdzf/a5AHTK+UxyVEBa0IQmOaltwKptnT7DM6rmjul0j7NrRja2LEjTpaU6IIBUwz1IlSu72k/P6yTkaDmp7Aw9LXwX/NygpWKdWovuLbx8YiYMwe2RUUm50XI4uYGrF8vJbmpsB///fuRa2SFj9YTvr7o7uKCZ2/2JFhzxUhlcicS2ysUuC7jMs2ekNsIgxAiibEbzlVnJdA33wAPPwy4ukpDPM7OMpJfafdrI4A7CqvkMVm3Dnj66eodi6neHL2VRSayuWrvZmzuAl7du/eaqk/uDQcr/iVsyb/ma7w6KjdXutPj9u1VJymZoy0/YgSwbh3g7a33cnWX8lpqhZclyf0/8mnHjphvYs5NdVdk1YdGcRddImrY1GqpB6S2N5xbtkz6+vTTUgACmJ84p129M3OmApkV0s83awbc0ADvvQeMGSP9kSyHqd6cnj2l+nQ0Cr2U9xVpOyTMpbyWO9FVW85UfXJvOFiTTJhyytU4vbe3t/TBf/EF8PTT0Fy+DBsZQw8apRI2zs5SQpvRow2WaawpyiuS+39EO8fF3GTYhhKAVBd7QgxgTwhR9fKOGJub8euv0iRXW1sp+VlAQPXaULn3IigI6N0b+Pdf4IEHgO+/B8z9LjfWm6NlY1Mpk6wJpo5Vr90yJ+HK+evVknVZTW4u8h99FB67d5scnhEALkVFocWmTVV6Pyqqi6Rm9a2657Uh9uYYwzwhRFRrcu+TY6qcthdk3LjqByDArfvpjBsnfQ0KAnbsAJycgJ9+Ap54AigvlwKmLVukrxX/2DbVm6Ol0QD33gt4eMjLsSKr3QoFVty8lUTlKqv716sl67Iab2+4R0RAbWYSj1qphPvdd5sMQAAg0s1NVr6OhpaivKLqntfGlv9DLgYhRGRQde6TY8g//wBffil9/+KLlmkTIGX5jo+XApRvvgFatZJ6bMaPl74GBUm9H4D8ux4vWQJ88IH0vaFruRDA8uXVmwgb7eWF2ZdDYZOvn0zNJl+F2ZerNxky2svLYonZrMXm22+hNDMco1SrYfPtt2brui0CM1T/vGqHz8b5+KCPu3uDPz45OBxjAIdjiMzfJweQejfS0w1fnH/8EZgwAQgLA3btsnz75syRelqMTZpdsEAaPklONl/X5s1Sb4uhuSPaOZIrVwIzZshvn24YSFFpousfblBoFDXKrmvpO/fWm+zsKtGqdlmuweW52dmAjOWmjWmIwpRGe16N4OqYWmIQQiQxdp8cQLo4V76QVp7D0aOHdJ+awEDLtku3pFZGL4ccFed6VD6GkyeB554DgoOBM2ekybG1bZ+hGx3KUV83Q7S4jRuByZN1T4VSiXInJ5x+/HF0/L//g+2VK1BU7CXZuBGYOFFW1bfbBfx2UK1rqKAqioqKBABRVFRk7aYQWd22bUIEBAghhSHSIzBQ2q61aZMQa9ZULRcQoF/OUvbs0d+PscdDDwnh5SWEQmH4dYVCOpbycuP7UquFWLJEiLw8y7dvz55b7ykvl55v3ix9rdwmQ+ehrj5fixs9Wggbm1snYsQIIXJypNdycqTn2hNiYyPEmDHWbS/VSnWuoQxCDGAQQqTP1AVy1y7TF3mFwvIXys2b5V3kN2+W9q1tR320rbrtE8J8gKE9hvr6fC3qxg0hXFykBru5CbF1q+FyW7dKrwNSeVORITVo1bmGcmIqEZlVeZVKxSGA9u2lJbiGaIdwZs3SX7VSW9WZNGvJux4LIb0nN9d0uRYt5NW3cSOwdKk05FV56ObCBWn7l1+aztcCWP7ztahr14A2baTEY2fOGM39gdGjpddHjADatgUM3CuHbj+cE2IA54QQyWeJfCLVZW7SrKE5F5aYT/HSS1LQMG6cNJnVkMxM6Tr666/Vq9sYZ2fg8mXz5Sz5+VZW689Ora7+5JdGMdmFDGGeECKqN5bIJ1JdSqWU7RSouqRW+zwuTv86Zqo3R65HHpGSm23ZAnz9ddX8JMnJQLduUgDi5GS8fQoF8MYbQFSU+X3KCUAAy36+FSUkSAGfsWXQslT3w2YA0mQwCCGiWqltPpGasuQwi1zduwMxMdL3I0fqX5i9vaXgJjcX6NpVuhPxtm3G2zd/PjB1quXaZunPF7i1OsrYUFG1AhEiAzgcYwCHY4jkq8nQiKX3X5/LVrdskQKPyrT5RO69F9i5E3B0NN8+uUNZXl7AxYvG87X4+Uk3B7TkcdfVMmO6/fEGdkRUb7RDI6NGVb1ZqrGhEUvvv67mQlSmVgNz5xp+TXtTv3/+Aeztb2031b7ISOlCbi6AW75cmrdp7Ga0Go3UA2PJ3hBz2WaFkAKf5GTjOVYaTR4TshoOxxBRrVljaMQaqnNhlkPu3JZRowx/vv7+UmLRnBxg4EBA5g13ddRq4/fdOX5cXh0//igFQRaZO0JNDodjDOBwDFHN3O5/CRsbiqlMmwZeLkPp4gMDpQDEVEbayEip5+Xee6VtvXpJKfK1Q0HV3WdAgDRh9sQJad83bshrv5cXkJdXdbs2kLqdAlEyj2nba4lBCBEZUpfLkWsTwP3xB3DffVJPSP/+wFdfAYcOGa9Ld18bM7/9VSqgwm1ZqnBykgINUyt4OHek6WEQUksMQojIEGtPwjXlwAFpyW9JCeDgIOUI0woIkIZ9oqPl3XfH1lYKVMrKpGXJgOG5PvHxQPPmwODB5ttXl3lMqGFhnhAiojpQk/wk9eXuu4EXX5S+rxiAALeW1G7aBLz/vvkb/5WXS0nSRo40P9dH7jyUuspjQo0bV8cQEVWDdhKuofkUledw1Ce1GvjoI8OvaXsxJkyQX582aIiOBoYNMz5UZK08MXR7YBBCRFRN5i7M1mBu5Y6W3DTwFYOG2iwzBgA3N6kcGXa7T+g2hcMxREQ1YIk08JYkd7hj7VopaKg8nKSlUEgrc+QGDaaGqLQKC4HYWPMTYZuipr60mUEIEdFtQO5wR8uWlp/XYixPTGAgMHas9P2CBVKiNyFM5yepLkvWVd+YFh+AaABWrVolWrduLVQqlejZs6c4ePCgyfJffPGF6NChg1CpVOKOO+4QO3bs0Ht90qRJAoDeY+DAgbLbU1RUJACIoqKiGh0PEVF9Ky8XIiBACIVCCOlSr/9QKIQIDJTKCSHEtm1S+YplAgOl7bVpw549QmzeLH3V7uvdd6X6bWyEeOutqvsNCKjZfg0dQ03rqm/a82XoXBk6X41Jda6hVg9CPv/8c2FnZyc++ugjcfLkSfHkk08KNzc3kZOTY7D8vn37hFKpFEuXLhV//vmneOWVV0SzZs3EiRMndGUmTZokBg0aJLKysnSPS5cuyW4TgxAiaoy2bZMuXpUDEe22yhdnY0FDXfjwQyGmTzccJBlrnynaY7VEXRXV12eyZ4/xAKTiY8+eutl/XWpUQUjPnj3F9OnTdc/VarXw9/cXsbGxBsuPHj1aPPjgg3rbIiIixNNPP617PmnSJDFs2LAat4lBCBE1VnXRw2EJlvzLv656EeqzZ2XzZnlByObNlt93XavONdSqc0LKyspw5MgRREVF6bbZ2NggKioKKSkpBt+TkpKiVx4ABg4cWKV8UlISvL290aFDBzz77LPIz8832o7S0lIUFxfrPYiIGqPoaCA9XUoOtnmz9DUtzfpp0y153x1L38MHqP/5GTVZ2tyY578YY9UluhcvXoRarYaPj4/edh8fH5w+fdrge7Kzsw2Wz87O1j0fNGgQoqOjERwcjHPnzuHll1/G4MGDkZKSAqWB2VaxsbFYvHixBY6IiMj66vPOwnLJXb0jp5wl6wKki/nMmYZX72jvjjxrlrQsu7qroIwtv42MlCbyXrhg+H3a7LuRkcDBg8C5c8BLL1XNTaPNhCtnnw3RbZknZKx2OjaALl26oGvXrmjbti2SkpLQr1+/KuXnz5+PmJgY3fPi4mIEBgbWS1uJiJoCuX/5e3hIX41dSK9elVLUy+HsfOt7Uxfm6vSsVCe4M3STQA8PKWvtqFHAypXSV+0+tCquUsrNBR54QDruyrS9NBVvEGjsxoSGgpWGwKrDMZ6enlAqlcjJydHbnpOTA19fX4Pv8fX1rVZ5AGjTpg08PT2Rmppq8HWVSgUXFxe9BxERWY42qZmxXCJaTz8NbNxoOHfGrFlAhw7Sxdvd3XxdTzwh3d3XVC6O/HwpwJCjOqnnjQ3vXLok3Y8nIcH40uaKafHPnzd+E0Ft4DJrlhRkVXdIqSEM71g1CLGzs0O3bt2QmJio26bRaJCYmIhevXoZfE+vXr30ygPArl27jJYHgMzMTOTn58OPeYOJiKxCzn13WrQAfH2BKVOqXkgzM6X3Z2YCrVtLwYqpuvz8pF6L5GTjF+aRIwF/f8DI6H8Vci8hpoZ3tLSBg7k5PNeumQ4OtL00SUnAjBnGh5Qq7hNoQEnS6mGirEmff/65UKlUYsOGDeLPP/8UTz31lHBzcxPZ2dlCCCEee+wxMW/ePF35ffv2CVtbW/HOO++IU6dOiUWLFukt0b18+bKYPXu2SElJEWlpaWL37t3irrvuEu3atRPXr1+X1SaujiEiqhumVu8UFQnh52d6tYirqxCXL5uv69o1IfLyTK+i0T7GjDGdYwUQwtZWiIwMecdoyeW3clfRvPKK/H3W1fJmrepcQ60+J2TMmDHIy8vDwoULkZ2djfDwcOzcuVM3+fT8+fOwsbnVYXPPPfdg8+bNeOWVV/Dyyy+jXbt2+Oqrr3DHHXcAAJRKJY4fP46NGzeisLAQ/v7+GDBgAJYsWQKVSmWVYyQiIomp++4kJZkf8igqAn79VerlMHcPnwMH5N1P5+mngdGjpR4ThaLq/AwhgPBwqddEy9QcE0tOnLV0B/6FC8C8eXUzCbdGahfv3J7YE0JEVP8snTujuvUZ61n56COpl0Zr61bD+US2bJFet2RPiNxMuLt3y9unNnutJdpmTKPqCSEiIgJqljvDkuXk3B152zZgzJiqdWRmSjcztLOT6jB1Z+GKy2/N0c6lMdZLA0iraPr0kbdPLy/z+wSqNwm3NngDOyIiahDMraCp7h1+a1Kfqbsjq9XA9Omm9/n889JXS94kUM4qGjkTf+PiqtZhTH2t42AQQkREDYLcC6nci7el60tOBipliKjiwgWpnJzAoTrkZMKVs09LB3q1pRDC1CKipqm4uBiurq4oKipizhAionpmKOFWYKAUMNQk4Zal6tuyRVrOas7mzVJPCmCd7KXm9qnNJwIYHt6pSZBUUXWuoQxCDGAQQkRkXZa+eFuivqQkKZ+GOXv2NLy0+ZVZOtCriEFILTEIISKiytRqKaGXucmfaWkN914tFdVVL011rqFcHUNERCSD3JUqjSEAARrGjQ45MZWIiEgmS084berYE0JERFQNcvKJkDwMQoiIiKqpIQxl3A44HENERERWwSCEiIiIrIJBCBEREVkFgxAiIiKyCgYhREREZBUMQoiIiMgquETXAG0m++LiYiu3hIiIqHHRXjvl3BWGQYgBly9fBgAEBgZauSVERESN0+XLl+Hq6mqyDG9gZ4BGo8G///4LZ2dnKLQ3BDCiuLgYgYGByMjI4M3urIznomHgeWgYeB4ajqZ2LoQQuHz5Mvz9/WFjY3rWB3tCDLCxsUFAQEC13uPi4tIk/nM1BjwXDQPPQ8PA89BwNKVzYa4HRIsTU4mIiMgqGIQQERGRVTAIqSWVSoVFixZBpVJZuylNHs9Fw8Dz0DDwPDQcPBfGcWIqERERWQV7QoiIiMgqGIQQERGRVTAIISIiIqtgEEJERERWwSCkllavXo2goCDY29sjIiIChw4dsnaTbms///wzhg4dCn9/fygUCnz11Vd6rwshsHDhQvj5+cHBwQFRUVE4e/asdRp7G4uNjUWPHj3g7OwMb29vDB8+HGfOnNErc/36dUyfPh0tWrSAk5MTRo4ciZycHCu1+Pa1du1adO3aVZcIq1evXvj+++91r/M8WMebb74JhUKBWbNm6bbxXFTFIKQWtm7dipiYGCxatAi//fYbwsLCMHDgQOTm5lq7abetkpIShIWFYfXq1QZfX7p0KVauXIl169bh4MGDcHR0xMCBA3H9+vV6buntbe/evZg+fToOHDiAXbt24caNGxgwYABKSkp0ZV544QV88803+PLLL7F37178+++/iI6OtmKrb08BAQF48803ceTIEfz666944IEHMGzYMJw8eRIAz4M1HD58GO+//z66du2qt53nwgBBNdazZ08xffp03XO1Wi38/f1FbGysFVvVdAAQ27dv1z3XaDTC19dXvP3227pthYWFQqVSiS1btlihhU1Hbm6uACD27t0rhJA+92bNmokvv/xSV+bUqVMCgEhJSbFWM5sMd3d38eGHH/I8WMHly5dFu3btxK5du8T9998vZs6cKYTgz4Qx7AmpobKyMhw5cgRRUVG6bTY2NoiKikJKSooVW9Z0paWlITs7W++cuLq6IiIiguekjhUVFQEAPDw8AABHjhzBjRs39M5Fx44d0apVK56LOqRWq/H555+jpKQEvXr14nmwgunTp+PBBx/U+8wB/kwYwxvY1dDFixehVqvh4+Ojt93HxwenT5+2UquatuzsbAAweE60r5HlaTQazJo1C71798Ydd9wBQDoXdnZ2cHNz0yvLc1E3Tpw4gV69euH69etwcnLC9u3b0blzZxw9epTnoR59/vnn+O2333D48OEqr/FnwjAGIURUK9OnT8cff/yBX375xdpNabI6dOiAo0ePoqioCPHx8Zg0aRL27t1r7WY1KRkZGZg5cyZ27doFe3t7azen0eBwTA15enpCqVRWmdmck5MDX19fK7WqadN+7jwn9ee5557Dt99+iz179iAgIEC33dfXF2VlZSgsLNQrz3NRN+zs7BASEoJu3bohNjYWYWFhWLFiBc9DPTpy5Ahyc3Nx1113wdbWFra2tti7dy9WrlwJW1tb+Pj48FwYwCCkhuzs7NCtWzckJibqtmk0GiQmJqJXr15WbFnTFRwcDF9fX71zUlxcjIMHD/KcWJgQAs899xy2b9+On376CcHBwXqvd+vWDc2aNdM7F2fOnMH58+d5LuqBRqNBaWkpz0M96tevH06cOIGjR4/qHt27d8ejjz6q+57noioOx9RCTEwMJk2ahO7du6Nnz56Ii4tDSUkJpkyZYu2m3bauXLmC1NRU3fO0tDQcPXoUHh4eaNWqFWbNmoXXX38d7dq1Q3BwMBYsWAB/f38MHz7ceo2+DU2fPh2bN2/G119/DWdnZ92YtqurKxwcHODq6orHH38cMTEx8PDwgIuLC2bMmIFevXrh7rvvtnLrby/z58/H4MGD0apVK1y+fBmbN29GUlISfvjhB56HeuTs7KybE6Xl6OiIFi1a6LbzXBhg7eU5jd17770nWrVqJezs7ETPnj3FgQMHrN2k29qePXsEgCqPSZMmCSGkZboLFiwQPj4+QqVSiX79+okzZ85Yt9G3IUPnAID4+OOPdWWuXbsmpk2bJtzd3UXz5s3FiBEjRFZWlvUafZuaOnWqaN26tbCzsxNeXl6iX79+4scff9S9zvNgPRWX6ArBc2GIQgghrBT/EBERURPGOSFERERkFQxCiIiIyCoYhBAREZFVMAghIiIiq2AQQkRERFbBIISIiIisgkEIERERWQWDECIiIrIKBiFE1CQkJSVBoVBUuYEYEVkPgxAiIiKyCgYhREREZBUMQoioXmg0GsTGxiI4OBgODg4ICwtDfHw8gFtDJTt27EDXrl1hb2+Pu+++G3/88YdeHdu2bUNoaChUKhWCgoKwbNkyvddLS0vx0ksvITAwECqVCiEhIfi///s/vTJHjhxB9+7d0bx5c9xzzz04c+ZM3R44ERnFIISI6kVsbCw++eQTrFu3DidPnsQLL7yACRMmYO/evboyc+bMwbJly3D48GF4eXlh6NChuHHjBgApeBg9ejTGjh2LEydO4NVXX8WCBQuwYcMG3fsnTpyILVu2YOXKlTh16hTef/99ODk56bXjP//5D5YtW4Zff/0Vtra2mDp1ar0cPxEZYO3b+BLR7e/69euiefPmYv/+/XrbH3/8cTFu3DixZ88eAUB8/vnnutfy8/OFg4OD2Lp1qxBCiPHjx4v+/fvrvX/OnDmic+fOQgghzpw5IwCIXbt2GWyDdh+7d+/WbduxY4cAIK5du2aR4ySi6mFPCBHVudTUVFy9ehX9+/eHk5OT7vHJJ5/g3LlzunK9evXSfe/h4YEOHTrg1KlTAIBTp06hd+/eevX27t0bZ8+ehVqtxtGjR6FUKnH//febbEvXrl113/v5+QEAcnNza32MRFR9ttZuABHd/q5cuQIA2LFjB1q2bKn3mkql0gtEasrBwUFWuWbNmum+VygUAKT5KkRU/9gTQkR1rnPnzlCpVDh//jxCQkL0HoGBgbpyBw4c0H1fUFCAv/76C506dQIAdOrUCfv27dOrd9++fWjfvj2USiW6dOkCjUajN8eEiBo29oQQUZ1zdnbG7Nmz8cILL0Cj0eDee+9FUVER9u3bBxcXF7Ru3RoA8Nprr6FFixbw8fHBf/7zH3h6emL48OEAgBdffBE9evTAkiVLMGbMGKSkpGDVqlVYs2YNACAoKAiTJk3C1KlTsXLlSoSFheGff/5Bbm4uRo8eba1DJyITGIQQUb1YsmQJvLy8EBsbi7///htubm6466678PLLL+uGQ958803MnDkTZ8+eRXh4OL755hvY2dkBAO666y588cUXWLhwIZYsWQI/Pz+89tprmDx5sm4fa9euxcsvv4xp06YhPz8frVq1wssvv2yNwyUiGRRCCGHtRhBR05aUlIS+ffuioKAAbm5u1m4OEdUTzgkhIiIiq2AQQkRERFbB4RgiIiKyCvaEEBERkVUwCCEiIiKrYBBCREREVsEghIiIiKyCQQgRERFZBYMQIiIisgoGIURERGQVDEKIiIjIKv4fZNtMsNUQMXQAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 600x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* background: */\n",
       "    progress::-webkit-progress-bar {background-color: #CDCDCD; width: 100%;}\n",
       "    progress {background-color: #CDCDCD;}\n",
       "\n",
       "    /* value: */\n",
       "    progress::-webkit-progress-value {background-color: #00BFFF  !important;}\n",
       "    progress::-moz-progress-bar {background-color: #00BFFF  !important;}\n",
       "    progress {color: #00BFFF ;}\n",
       "\n",
       "    /* optional */\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #000000;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      <progress value='44' class='progress-bar-interrupted' max='100' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      44.00% [44/100 2:43:19<3:27:51][earlystopping]\n",
       "      <br>\n",
       "      \n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;31m<<<<<< val_loss without improvement in 10 epoch,early stopping >>>>>>\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>lr</th>\n",
       "      <th>val_loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0.384217</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.249804</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>0.251601</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.205617</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>0.219351</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.168795</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>0.173330</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.160446</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0.147875</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.135188</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>0.138851</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.133749</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>7</td>\n",
       "      <td>0.140442</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.125064</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>0.127080</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.121878</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>9</td>\n",
       "      <td>0.129889</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.118764</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10</td>\n",
       "      <td>0.133962</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.112397</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>11</td>\n",
       "      <td>0.109882</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.114135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>12</td>\n",
       "      <td>0.116118</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.111593</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>13</td>\n",
       "      <td>0.095580</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.110649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>14</td>\n",
       "      <td>0.099505</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.115588</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>15</td>\n",
       "      <td>0.103086</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.113059</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>16</td>\n",
       "      <td>0.108844</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.108111</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>17</td>\n",
       "      <td>0.094075</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.104764</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>18</td>\n",
       "      <td>0.095604</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.100785</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>19</td>\n",
       "      <td>0.081102</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.097781</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>20</td>\n",
       "      <td>0.070968</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.101562</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>21</td>\n",
       "      <td>0.097898</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.090838</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>22</td>\n",
       "      <td>0.088633</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.090873</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>23</td>\n",
       "      <td>0.074751</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.095917</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>24</td>\n",
       "      <td>0.079089</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.101793</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>25</td>\n",
       "      <td>0.084095</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.091652</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>26</td>\n",
       "      <td>0.065277</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.089520</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>27</td>\n",
       "      <td>0.074971</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.085294</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>28</td>\n",
       "      <td>0.069898</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.089764</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>29</td>\n",
       "      <td>0.071201</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.084973</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>30</td>\n",
       "      <td>0.058163</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.090691</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>31</td>\n",
       "      <td>0.060202</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.091671</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>32</td>\n",
       "      <td>0.066866</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.089982</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>33</td>\n",
       "      <td>0.062126</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.092677</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>34</td>\n",
       "      <td>0.050651</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.083426</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>35</td>\n",
       "      <td>0.043467</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.094646</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>36</td>\n",
       "      <td>0.050430</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.083600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>37</td>\n",
       "      <td>0.042841</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.089836</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>38</td>\n",
       "      <td>0.048359</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.093890</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>39</td>\n",
       "      <td>0.038404</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.093853</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>40</td>\n",
       "      <td>0.044234</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.093282</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>41</td>\n",
       "      <td>0.053240</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.085225</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>42</td>\n",
       "      <td>0.045331</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.087440</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>43</td>\n",
       "      <td>0.043789</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.097614</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>44</td>\n",
       "      <td>0.041338</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.103412</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    epoch  train_loss       lr  val_loss\n",
       "0       1    0.384217  0.00006  0.249804\n",
       "1       2    0.251601  0.00006  0.205617\n",
       "2       3    0.219351  0.00006  0.168795\n",
       "3       4    0.173330  0.00006  0.160446\n",
       "4       5    0.147875  0.00006  0.135188\n",
       "5       6    0.138851  0.00006  0.133749\n",
       "6       7    0.140442  0.00006  0.125064\n",
       "7       8    0.127080  0.00006  0.121878\n",
       "8       9    0.129889  0.00006  0.118764\n",
       "9      10    0.133962  0.00006  0.112397\n",
       "10     11    0.109882  0.00006  0.114135\n",
       "11     12    0.116118  0.00006  0.111593\n",
       "12     13    0.095580  0.00006  0.110649\n",
       "13     14    0.099505  0.00006  0.115588\n",
       "14     15    0.103086  0.00006  0.113059\n",
       "15     16    0.108844  0.00006  0.108111\n",
       "16     17    0.094075  0.00006  0.104764\n",
       "17     18    0.095604  0.00006  0.100785\n",
       "18     19    0.081102  0.00006  0.097781\n",
       "19     20    0.070968  0.00006  0.101562\n",
       "20     21    0.097898  0.00006  0.090838\n",
       "21     22    0.088633  0.00006  0.090873\n",
       "22     23    0.074751  0.00006  0.095917\n",
       "23     24    0.079089  0.00006  0.101793\n",
       "24     25    0.084095  0.00006  0.091652\n",
       "25     26    0.065277  0.00006  0.089520\n",
       "26     27    0.074971  0.00006  0.085294\n",
       "27     28    0.069898  0.00006  0.089764\n",
       "28     29    0.071201  0.00006  0.084973\n",
       "29     30    0.058163  0.00006  0.090691\n",
       "30     31    0.060202  0.00006  0.091671\n",
       "31     32    0.066866  0.00006  0.089982\n",
       "32     33    0.062126  0.00006  0.092677\n",
       "33     34    0.050651  0.00006  0.083426\n",
       "34     35    0.043467  0.00006  0.094646\n",
       "35     36    0.050430  0.00006  0.083600\n",
       "36     37    0.042841  0.00006  0.089836\n",
       "37     38    0.048359  0.00006  0.093890\n",
       "38     39    0.038404  0.00006  0.093853\n",
       "39     40    0.044234  0.00006  0.093282\n",
       "40     41    0.053240  0.00006  0.085225\n",
       "41     42    0.045331  0.00006  0.087440\n",
       "42     43    0.043789  0.00006  0.097614\n",
       "43     44    0.041338  0.00006  0.103412"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# keras_model.load_ckpt(ckpt_path) #支持加载微调后的权重继续训练(断点续训)\n",
    "keras_model.fit(train_data = dl_train,\n",
    "                val_data = dl_val,\n",
    "                epochs=100,patience=10,\n",
    "                monitor='val_loss',mode='min',\n",
    "                ckpt_path = ckpt_path\n",
    "               )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "108976a7-17cc-411e-8532-d781d65d1f29",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "82a55a90-f980-4dbd-b8f8-6c0e0c8d5a5b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "455498a9-3613-4efa-b7d4-6035314c8fdc",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "49763c6a-6ee5-448c-b873-48825a6fba67",
   "metadata": {},
   "source": [
    "## 四，保存模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f98c4e5-dbb0-456e-bc4b-1c8dfccbd499",
   "metadata": {},
   "source": [
    "为减少GPU压力，此处可重启kernel释放显存"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ee686c58-42e2-463a-a429-3fe16dee5a7b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings \n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "142a917e-9db0-4d5a-8352-cb250561f47d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2023-07-22 23:30:48,987] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The model weights are not tied. Please use the `tie_weights` method before using the `infer_auto_device` function.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6741a4b4d00d4f2f864159711ec34b10",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import torch\n",
    "from transformers import AutoTokenizer, AutoModelForCausalLM,AutoConfig, AutoModel, BitsAndBytesConfig\n",
    "from transformers.generation.utils import GenerationConfig\n",
    "import torch.nn as nn\n",
    "model_name_or_path ='../baichuan-13b'\n",
    "ckpt_path = 'baichuan13b_ner'\n",
    "tokenizer = AutoTokenizer.from_pretrained(\n",
    "    model_name_or_path,\n",
    "    trust_remote_code=True\n",
    ")\n",
    "model_old = AutoModelForCausalLM.from_pretrained(\n",
    "    model_name_or_path,\n",
    "    trust_remote_code=True,\n",
    "    low_cpu_mem_usage=True,\n",
    "    torch_dtype=torch.float16,\n",
    "    device_map='auto'\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2a8b409f-f100-48c5-9794-62cb5bdf19d4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "===================================BUG REPORT===================================\n",
      "Welcome to bitsandbytes. For bug reports, please run\n",
      "\n",
      "python -m bitsandbytes\n",
      "\n",
      " and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues\n",
      "================================================================================\n",
      "bin /usr/local/lib/python3.8/dist-packages/bitsandbytes/libbitsandbytes_cuda116.so\n",
      "CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...\n",
      "CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so.11.0\n",
      "CUDA SETUP: Highest compute capability among GPUs detected: 8.0\n",
      "CUDA SETUP: Detected CUDA version 116\n",
      "CUDA SETUP: Loading binary /usr/local/lib/python3.8/dist-packages/bitsandbytes/libbitsandbytes_cuda116.so...\n"
     ]
    }
   ],
   "source": [
    "from peft import PeftModel\n",
    "\n",
    "#可能需要5分钟左右\n",
    "peft_model = PeftModel.from_pretrained(model_old, ckpt_path)\n",
    "model_new = peft_model.merge_and_unload()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "2c060b4a-7731-4ec7-8699-9f47088f4a2f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers.generation.utils import GenerationConfig\n",
    "model_new.generation_config = GenerationConfig.from_pretrained(model_name_or_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca83a654-1d18-47e7-bbf1-d67ca351813d",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "524a3d55-5daf-4ff1-8fb9-e3dd60d65f18",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "乔戈里峰。世界第二高峰———乔戈里峰西方登山者称其为k2峰，海拔高度是8611米，位于喀喇昆仑山脉的中巴边境上.\n"
     ]
    }
   ],
   "source": [
    "from IPython.display import clear_output\n",
    "messages = []\n",
    "messages.append({\"role\": \"user\",\n",
    "                 \"content\": \"世界上第二高的山峰是什么？\"})\n",
    "response = model_new.chat(tokenizer,messages=messages,stream=True)\n",
    "for res in response:\n",
    "    print(res)\n",
    "    clear_output(wait=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "ce165cd8-987d-47be-95ac-83cf7ef479b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "save_path = 'baichuan-13b-ner'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "9214d354-811b-4e47-96ac-682ed5e68ee9",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer.save_pretrained(save_path)\n",
    "model_new.save_pretrained(save_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "a7950603-e844-4d60-990d-f86245fbaa26",
   "metadata": {},
   "outputs": [],
   "source": [
    "!cp ../baichuan-13b/*.py  baichuan-13b-ner"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a91c249-3402-489b-ae8e-5554a5082562",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "81582c3e-28d3-4847-a84c-dc1b53aa0839",
   "metadata": {},
   "source": [
    "## 五，使用模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52c57c2f-0e1a-4e77-b4e4-fefa210b1bb1",
   "metadata": {},
   "source": [
    "为减少GPU压力，此处可再次重启kernel释放显存。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "2bec4773-61f0-4aa2-9ac2-d2e1c295b2e3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2023-07-23 00:02:25,962] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)\n",
      "\n",
      "===================================BUG REPORT===================================\n",
      "Welcome to bitsandbytes. For bug reports, please run\n",
      "\n",
      "python -m bitsandbytes\n",
      "\n",
      " and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues\n",
      "================================================================================\n",
      "bin /usr/local/lib/python3.8/dist-packages/bitsandbytes/libbitsandbytes_cuda116.so\n",
      "CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...\n",
      "CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so\n",
      "CUDA SETUP: Highest compute capability among GPUs detected: 8.0\n",
      "CUDA SETUP: Detected CUDA version 116\n",
      "CUDA SETUP: Loading binary /usr/local/lib/python3.8/dist-packages/bitsandbytes/libbitsandbytes_cuda116.so...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "188e4f99126d4482844dc08ff6506e9b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import torch\n",
    "from transformers import AutoTokenizer, AutoModelForCausalLM,AutoConfig, BitsAndBytesConfig\n",
    "from transformers.generation.utils import GenerationConfig\n",
    "import torch.nn as nn\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "model_name_or_path = 'baichuan-13b-ner'\n",
    "\n",
    "bnb_config=BitsAndBytesConfig(\n",
    "            load_in_4bit=True,\n",
    "            bnb_4bit_compute_dtype=torch.float16,\n",
    "            bnb_4bit_use_double_quant=True,\n",
    "            bnb_4bit_quant_type=\"nf4\",\n",
    "            llm_int8_threshold=6.0,\n",
    "            llm_int8_has_fp16_weight=False,\n",
    "        )\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(\n",
    "   model_name_or_path, trust_remote_code=True)\n",
    "\n",
    "model = AutoModelForCausalLM.from_pretrained(model_name_or_path,\n",
    "                quantization_config=bnb_config,\n",
    "                trust_remote_code=True) \n",
    "\n",
    "model.generation_config = GenerationConfig.from_pretrained(model_name_or_path)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e958b920-63cd-4e1a-bd28-d36e7c281901",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "乔戈里峰。世界第二高峰———乔戈里峰海拔高度：8610米登山难度：死亡率：5.\n"
     ]
    }
   ],
   "source": [
    "from IPython.display import clear_output\n",
    "messages = []\n",
    "messages.append({\"role\": \"user\",\n",
    "                 \"content\": \"世界上第二高的山峰是什么？\"})\n",
    "response = model.chat(tokenizer,messages=messages,stream=True)\n",
    "for res in response:\n",
    "    print(res)\n",
    "    clear_output(wait=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e63d6988-4cfd-46ba-8a5d-f11e5d183483",
   "metadata": {},
   "source": [
    "我们测试一下微调后的效果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b8e7ad1b-9831-4b11-9ef8-a848a49c9882",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd \n",
    "import numpy as np \n",
    "import datasets \n",
    "from tqdm import tqdm \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "57a77d28-11ab-423c-8856-c4027d2e9faf",
   "metadata": {},
   "outputs": [],
   "source": [
    "prefix = '''命名实体识别：抽取文本中的 人名，地点，组织 这三类命名实体，并按照json格式返回结果。\n",
    "\n",
    "下面是一些范例：\n",
    "\n",
    "小明对小红说:\"你听说过安利吗？\" -> {\"人名\": [\"小明\",\"小红\"], \"组织\": [\"安利\"]}\n",
    "现在，每年有几十万中国人到美国访问，几千名中国留学生到美国就学。 -> {\"地点\": [\"中国\", \"美国\"]}\n",
    "中国是联合国安理会常任理事国之一。 -> {\"地点\": [\"中国\"], \"组织\": [\"联合国\"]}\n",
    "\n",
    "请对下述文本进行实体抽取，返回json格式。\n",
    "\n",
    "'''\n",
    "\n",
    "def get_prompt(text):\n",
    "    return prefix+text+' -> '\n",
    "\n",
    "def get_message(prompt,response):\n",
    "    return [{\"role\": \"user\", \"content\": f'{prompt} -> '},\n",
    "            {\"role\": \"assistant\", \"content\": response}]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "7e946989-aa53-4e66-b2fa-decbcb708711",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'人名': ['摩洛哥']}\n"
     ]
    }
   ],
   "source": [
    "messages  = [{\"role\": \"user\", \"content\": get_prompt(\"一些摩洛哥球迷已按捺不住，在看台上欢呼雀跃\")}]\n",
    "response = model.chat(tokenizer, messages)\n",
    "print(response)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "a60b335c-5488-4907-a7c5-3f327db98ab8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'role': 'user',\n",
       "  'content': '命名实体识别：抽取文本中的 人名，地点，组织 这三类命名实体，并按照json格式返回结果。\\n\\n下面是一些范例：\\n\\n小明对小红说:\"你听说过安利吗？\" -> {\"人名\": [\"小明\",\"小红\"], \"组织\": [\"安利\"]}\\n现在，每年有几十万中国人到美国访问，几千名中国留学生到美国就学。 -> {\"地点\": [\"中国\", \"美国\"]}\\n中国是联合国安理会常任理事国之一。 -> {\"地点\": [\"中国\"], \"组织\": [\"联合国\"]}\\n\\n请对下述文本进行实体抽取，返回json格式。\\n\\n一些摩洛哥球迷已按捺不住，在看台上欢呼雀跃 -> '},\n",
       " {'role': 'assistant', 'content': \"{'地点': ['摩洛哥']}\"},\n",
       " {'role': 'user', 'content': '这次轮到北京国安队，不知会不会再步后尘？ -> '},\n",
       " {'role': 'assistant', 'content': \"{'组织': ['北京国安队']}\"},\n",
       " {'role': 'user', 'content': '革命党人孙中山在澳门成立同盟会分会 -> '},\n",
       " {'role': 'assistant',\n",
       "  'content': \"{'人名': ['孙中山'], '地名': ['澳门'], '组织': ['同盟会']}\"},\n",
       " {'role': 'user', 'content': '我曾在安徽芜湖市和上海浦东打工。 -> '},\n",
       " {'role': 'assistant', 'content': \"{'地点': ['安徽芜湖市', '上海浦东']}\"}]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "messages = messages+[{\"role\": \"assistant\", \"content\": \"{'地点': ['摩洛哥']}\"}]\n",
    "messages.extend(get_message(\"这次轮到北京国安队，不知会不会再步后尘？\",\"{'组织': ['北京国安队']}\"))\n",
    "messages.extend(get_message(\"革命党人孙中山在澳门成立同盟会分会\",\"{'人名': ['孙中山'], '地名': ['澳门'], '组织': ['同盟会']}\"))\n",
    "messages.extend(get_message(\"我曾在安徽芜湖市和上海浦东打工。\",\"{'地点': ['安徽芜湖市', '上海浦东']}\"))\n",
    "display(messages)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "7c1a9fb1-2949-468d-84d7-56c2fd3d67d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict(text,temperature=0.01):\n",
    "    model.generation_config.temperature=temperature\n",
    "    response = model.chat(tokenizer, \n",
    "                          messages = messages+[{'role':'user','content':f'{text} -> '}])\n",
    "    return response\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "1cc4a47b-d2a1-4009-adbd-a357b790bf27",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "import pandas as pd \n",
    "\n",
    "df = pd.read_pickle('dfner_13k.pkl')\n",
    "dfdata,dftest = train_test_split(df,test_size=300,random_state=42)\n",
    "dftrain,dfval = train_test_split(dfdata,test_size=200,random_state=42)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "b00718e0-a573-437e-8844-83374fae042f",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 300/300 [10:42<00:00,  2.14s/it]  \n"
     ]
    }
   ],
   "source": [
    "preds = ['' for x in dftest['target']]\n",
    "for i in tqdm(range(len(preds))):\n",
    "    preds[i] = predict(dftest['text'].iloc[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "aa488a58-77c0-4cd8-b71b-3bc0a9236439",
   "metadata": {},
   "outputs": [],
   "source": [
    "def toset(s):\n",
    "    try:\n",
    "        dic = eval(str(s))\n",
    "        res = []\n",
    "        for k,v in dic.items():\n",
    "            for x in v:\n",
    "                if x:\n",
    "                    res.append((k,x))\n",
    "        return set(res)\n",
    "    except Exception as err:\n",
    "        print(err)\n",
    "        return set()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "1fd4eff2-a7e8-4356-ba0f-2b75945e8212",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EOL while scanning string literal (<string>, line 1)\n",
      "precision = 0.9139280125195618\n",
      "recall = 0.8427128427128427\n",
      "f1_score = 0.876876876876877\n"
     ]
    }
   ],
   "source": [
    "dftest['pred'] = [toset(x) for x in preds]\n",
    "dftest['gt'] = [toset(x) for x in dftest['target']]\n",
    "dftest['tp_cnt'] = [len(pred&gt) for pred,gt in zip(dftest['pred'],dftest['gt'])]\n",
    "dftest['pred_cnt'] = [len(x) for x in dftest['pred']]\n",
    "dftest['gt_cnt'] = [len(x) for x in dftest['gt']]\n",
    "\n",
    "precision = sum(dftest['tp_cnt'])/sum(dftest['pred_cnt'])\n",
    "print('precision = '+str(precision))\n",
    "\n",
    "recall = sum(dftest['tp_cnt'])/sum(dftest['gt_cnt'])\n",
    "print('recall = '+str(recall))\n",
    "\n",
    "f1 = 2*precision*recall/(precision+recall)\n",
    "print('f1_score = '+str(f1))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e61d9def-fb3b-4bf1-8579-b5e28415082c",
   "metadata": {},
   "source": [
    "微调后的f1_score为0.8768，相比微调前的f1_score=0.44，取得了不可忽视的巨大提升。\n",
    "\n",
    "以上。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb39f77a-af07-450f-bead-b69a3bffa06f",
   "metadata": {},
   "source": [
    "**如果本项目对你有所帮助，想鼓励一下作者，记得给本项目加一颗星星star⭐️，并分享给你的朋友们喔😊!** \n",
    "\n",
    "如果在torchkeras的使用中遇到问题，可以在项目中提交issue。\n",
    "\n",
    "如果想要获得更快的反馈或者与其他torchkeras用户小伙伴进行交流，\n",
    "\n",
    "可以在公众号算法美食屋后台回复关键字：**加群**。\n",
    "\n",
    "![](https://tva1.sinaimg.cn/large/e6c9d24egy1h41m2zugguj20k00b9q46.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a05d7b5a-effc-475e-802d-68c831186d6d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32a99895-12d1-4ec0-9c53-e13d3d621561",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}