TxMM Lecture 9: Extraction Pipeline

Guest lecture by Klaus Lux, where we go through an example pipeline using sklearn.

UCI YouTube Spam Collection Data Set.

Preprocessing

Setting up

  • Set up Git repo
  • Make a requirements.txt file
    • Include all Python packages your project relies on
    • pip install -r requirements.txt
  • Use virtual environments, e.g. using venv or pyenv to have the right dependencies per project
    • E.g. a new virtual environment for each large project
    • Q: Does Python duplicate packages or re-use resources when possible?
  • IDEs like PyCharm has support for git and also virtual environments.

Download dataset

  • Take a quick visual glance on data
  • Idea: have data loading, requests, making train_test splits etc. in a separate file, e.g. loading.py
    • sklearn: train_test_split()

Preprocessing

  • Build feature space, e.g. using CountVectorizer()
  • etc.

Fit, transform, predict

  • clf = X.fit_transform() (or separately)
  • clf.predict(validation_set)

Evaluation:

  • Take into account characteristics of your data and your priorities to choose a
  • accuracy_score if data is very balanced and in simple cases
  • precision, recall, F1 etc.
  • ROC curve for binary tasks to check calibration
    • Outputting probability estimates is more useful here, because then we can apply multiple thresholds and check which decision is optimal
    • Thresholding shows us possible trade-offs between classifying false positives and true positives
  • Tip for quick testing: in PyCharm you can stop at a breakpoint and then open an interactive terminal in that state.
  • Tip: if you need more custom eval, make a separate utility test.

Sklearn pipelines with hyperparameters

Pipeline takes a list of estimators in (name, class_instance) tuples.

E.g. p = Pipeline([('vect', 'CountVectorizer()'), ('cf', MultinomialNB()])

Then as usual:

p.fit(train_text, train_labels)

p.fit()

Grid search

Get the tuning right so you can focus on your research question.

For hyperparameter tuning this also allows us to quickly incorporate grid search on the pipeline.

parameters = { 'vect__ngram_range': [ (1,1), (1,2), (1,3)], 'clf__alpha': [0, 0.5, 1] }

gc_p = GridSearchCV(p, parameters, cv=5, n_jobs=-1, verbose=10, scoring='f1')

gc_p.fit(train_data)

(n_jobs=-1; multi-threading?)

print(gc_p.best_score_) print(gc_p.best_params_)

Visualisation

There is also a nice visualization option for parameters.

  • Read results into dataframe
  • Make colored grid
  • Plot with plotly: interactive parallel coordinates plot

Implementing your own classifier to play nice with sklearn

You can also plug in your own classifiers if sklearn doesn't have it by default. It is quite easy to fit the sklearn interface.

Other

  • TODO note to self: Check out debugger tools; en tools that record stored variables and values.

Assignment feedback

W.r.t. assignment 2:

  • Work towards and comply with a standard; important for interoperationality
    • You want to understand output; avoid nasty surprises

Ass 4Failure analysis:

  • Inspect mistakes of classifier
    • Which data points
    • Sample data points 100 test cases
  • Failure analysis at the very end: if you use any feedback from failure analysis, you need to use a new unseen test set next time