超参数调整

详细可以参考官方文档

在拟合模型之前需要定义好的参数

Linear regression: Choosing parameters
Ridge/lasso regression: Choosing alpha
k-Nearest Neighbors: Choosing n_neighbors
Parameters like alpha and k: Hyperparameters
Hyperparameters cannot be learned by tting the model

sklearn.model_selection.GridSearchCV

超参数自动搜索模块
网格搜索+交叉验证
指定的参数范围内，按步长依次调整参数，利用调整的参数训练学习器，从所有的参数中找到在验证集上精度最高的参数，这其实是一个训练和比较的过程

class sklearn.model_selection.GridSearchCV(estimator, param_grid, scoring=None, n_jobs=None, iid='deprecated', refit=True, cv=None, verbose=0, pre_dispatch='2*n_jobs', error_score=nan, return_train_score=False

参数

estimator:模型对象
param_grid：dict or list of dictionaries，字典类型的参数，定义一个字典然后都放进去
scoring：string, callable, list/tuple, dict or None, default: None，就是metrics，损失函数定义rmse，mse等
- none的话就是默认estimator的score
- 定义规则
  - score规则
  - 自定义score
Number of jobs to run in parallel. None means 1 unless in a joblib.parallel_backend context. -1 means using all processors. See Glossary for more details.控制cop,core并行运行数量

-cv：int, cross-validation generator or an iterable, optional，k折交叉验证数，默认5折
- Determines the cross-validation splitting strategy. Possible inputs for cv are:
- None, to use the default 5-fold cross validation,integer, to specify the number of folds in a (Stratified)KFold,CV splitter,
- An iterable yielding (train, test) splits as arrays of indices.
- For integer/None inputs, if the estimator is a classifier and y is either binary or multiclass, StratifiedKFold is used. In all other cases, KFold is used.
verbose：控制输出信息的详细程度，愈高输出越多。

属性

常见：

cv_results_dict of numpy (masked) ndarrays输出交叉验证的每一个结果
best_estimator_:最好的估计器
best_params_：dict
- 返回最优模型参数
- Parameter setting that gave the best results on the hold out data.
- For multi-metric evaluation, this is present only if refit is specified.
best_score_：float
- 返回最优模型参数的得分
- Mean cross-validated score of the best_estimator
- For multi-metric evaluation, this is present only if refit is specified.
- This attribute is not available if refit is a function.
Import necessary modules

from sklearn.model_selection import GridSearchCV

from sklearn.linear_model import LogisticRegression

Setup the hyperparameter grid

创建一个参数集

c_space = np.logspace(-5, 8, 15)

这里是创建一个字典保存参数集

param_grid = {'C': c_space}

Instantiate a logistic regression classifier: logreg

针对回归模型进行的超参数调整

logreg = LogisticRegression()

Instantiate the GridSearchCV object: logreg_cv

logreg_cv = GridSearchCV(logreg, param_grid, cv=5)

Fit it to the data

logreg_cv.fit(X,y)

Print the tuned parameters and score

得到最好的模型

print("Tuned Logistic Regression Parameters: {}".format(logreg_cv.best_params_))

得到最好的模型的最好的结果

print("Best score is {}".format(logreg_cv.best_score_))

output:
Tuned Logistic Regression Parameters: {'C': 3.727593720314938}
Best score is 0.7708333333333334

GridSearchCV can be computationally expensive, especially if you are searching over a large hyperparameter space and dealing with multiple hyperparameters. A solution to this is to use RandomizedSearchCV, in which not all hyperparameter values are tried out. Instead, a fixed number of hyperparameter settings is sampled from specified probability distributions.

grid相当于一个for循环，会遍历每一个参数，因此，当调参很多的时候，会导致计算量非常的大，因此，使用随机抽样的随机搜索会好一些

RandomizedSearchCV的使用方法其实是和GridSearchCV一致的，但它以随机在参数空间中采样的方式代替了GridSearchCV对于参数的网格搜索，在对于有连续变量的参数时，RandomizedSearchCV会将其当作一个分布进行采样这是网格搜索做不到的，它的搜索能力取决于设定的n_iter参数，同样的给出代码

csdn

RandomizedSearchCV

随机搜索法
不是每一个参数都被选取，而是从指定概率分布的参数中，抽取一定量的参数

我还是没太能明白？

可以比较一下时间

比较网格搜索而言，参数略有不同

算了，还是都列一下常见的吧，剩下的可以参照官方文档

比Grid 多了一个属性

.cv_results_，可以交叉验证的每一轮的结果

Import necessary modules

from scipy.stats import randint
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import RandomizedSearchCV

Setup the parameters and distributions to sample from: param_dist

以决策树为例，注意定一个字典的形式哦

param_dist = {"max_depth": [3, None],
"max_features": randint(1, 9),
"min_samples_leaf": randint(1, 9),
"criterion": ["gini", "entropy"]}

Instantiate a Decision Tree classifier: tree

tree = DecisionTreeClassifier()

Instantiate the RandomizedSearchCV object: tree_cv

tree_cv = RandomizedSearchCV(tree, param_dist, cv=5)

Fit it to the data

tree_cv.fit(X,y)

Print the tuned parameters and score

print("Tuned Decision Tree Parameters: {}".format(tree_cv.best_params_))
print("Best score is {}".format(tree_cv.best_score_))

output:
Tuned Decision Tree Parameters: {'criterion': 'gini', 'max_depth': 3, 'max_features': 5, 'min_samples_leaf': 2}
Best score is 0.7395833333333334