{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n\n# Specify Hyperparameters Manually\n\nIt's natural that you have some specific sets of hyperparameters to try first such as initial learning rate\nvalues and the number of leaves.\nAlso, it's also possible that you've already tried those sets before having Optuna find better\nsets of hyperparameters.\n\nOptuna provides two APIs to support such cases:\n\n1. Passing those sets of hyperparameters and let Optuna evaluate them - :func:`~optuna.study.Study.enqueue_trial`\n2. Adding the results of those sets as completed ``Trial``\\s - :func:`~optuna.study.Study.add_trial`\n\n## First Scenario: Have Optuna evaluate your hyperparameters\n\nIn this scenario, let's assume you have some out-of-box sets of hyperparameters but have not\nevaluated them yet and decided to use Optuna to find better sets of hyperparameters.\n\nOptuna has :func:`optuna.study.Study.enqueue_trial` which lets you pass those sets of\nhyperparameters to Optuna and Optuna will evaluate them.\n\nThis section walks you through how to use this lit API with [LightGBM](https://lightgbm.readthedocs.io/en/latest/).\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import lightgbm as lgb\nimport numpy as np\nimport sklearn.datasets\nimport sklearn.metrics\nfrom sklearn.model_selection import train_test_split\n\nimport optuna"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Define the objective function.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "def objective(trial):\n    data, target = sklearn.datasets.load_breast_cancer(return_X_y=True)\n    train_x, valid_x, train_y, valid_y = train_test_split(data, target, test_size=0.25)\n    dtrain = lgb.Dataset(train_x, label=train_y)\n    dvalid = lgb.Dataset(valid_x, label=valid_y)\n\n    param = {\n        \"objective\": \"binary\",\n        \"metric\": \"auc\",\n        \"verbosity\": -1,\n        \"boosting_type\": \"gbdt\",\n        \"bagging_fraction\": min(trial.suggest_float(\"bagging_fraction\", 0.4, 1.0 + 1e-12), 1),\n        \"bagging_freq\": trial.suggest_int(\"bagging_freq\", 0, 7),\n        \"min_child_samples\": trial.suggest_int(\"min_child_samples\", 5, 100),\n    }\n\n    # Add a callback for pruning.\n    pruning_callback = optuna.integration.LightGBMPruningCallback(trial, \"auc\")\n    gbm = lgb.train(\n        param, dtrain, valid_sets=[dvalid], verbose_eval=False, callbacks=[pruning_callback]\n    )\n\n    preds = gbm.predict(valid_x)\n    pred_labels = np.rint(preds)\n    accuracy = sklearn.metrics.accuracy_score(valid_y, pred_labels)\n    return accuracy"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Then, construct ``Study`` for hyperparameter optimization.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "study = optuna.create_study(direction=\"maximize\", pruner=optuna.pruners.MedianPruner())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Here, we get Optuna evaluate some sets with larger ``\"bagging_fraq\"`` value and\nthe default values.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "study.enqueue_trial(\n    {\n        \"bagging_fraction\": 1.0,\n        \"bagging_freq\": 0,\n        \"min_child_samples\": 20,\n    }\n)\n\nstudy.enqueue_trial(\n    {\n        \"bagging_fraction\": 0.75,\n        \"bagging_freq\": 5,\n        \"min_child_samples\": 20,\n    }\n)\n\nimport logging\nimport sys\n\n# Add stream handler of stdout to show the messages to see Optuna works expectedly.\noptuna.logging.get_logger(\"optuna\").addHandler(logging.StreamHandler(sys.stdout))\nstudy.optimize(objective, n_trials=100, timeout=600)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Second scenario: Have Optuna utilize already evaluated hyperparameters\n\nIn this scenario, let's assume you have some out-of-box sets of hyperparameters and\nyou have already evaluated them but the results are not desirable so that you are thinking of\nusing Optuna.\n\nOptuna has :func:`optuna.study.Study.add_trial` which lets you register those results\nto Optuna and then Optuna will sample hyperparameters taking them into account.\n\nIn this section,  the ``objective`` is the same as the first scenario.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "study = optuna.create_study(direction=\"maximize\", pruner=optuna.pruners.MedianPruner())\nstudy.add_trial(\n    optuna.trial.create_trial(\n        params={\n            \"bagging_fraction\": 1.0,\n            \"bagging_freq\": 0,\n            \"min_child_samples\": 20,\n        },\n        distributions={\n            \"bagging_fraction\": optuna.distributions.UniformDistribution(0.4, 1.0 + 1e-12),\n            \"bagging_freq\": optuna.distributions.IntUniformDistribution(0, 7),\n            \"min_child_samples\": optuna.distributions.IntUniformDistribution(5, 100),\n        },\n        value=0.94,\n    )\n)\nstudy.add_trial(\n    optuna.trial.create_trial(\n        params={\n            \"bagging_fraction\": 0.75,\n            \"bagging_freq\": 5,\n            \"min_child_samples\": 20,\n        },\n        distributions={\n            \"bagging_fraction\": optuna.distributions.UniformDistribution(0.4, 1.0 + 1e-12),\n            \"bagging_freq\": optuna.distributions.IntUniformDistribution(0, 7),\n            \"min_child_samples\": optuna.distributions.IntUniformDistribution(5, 100),\n        },\n        value=0.95,\n    )\n)\nstudy.optimize(objective, n_trials=100, timeout=600)"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.6"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}