Learning Apache Spark

Ming Chen & Wenqiang Feng

Introduction

This repository contains mainly notes from learning Apache Spark by Ming Chen & Wenqiang Feng. We try to use the detailed demo code and examples to show how to use pyspark for big data mining. If you find your work wasn’t cited in this note, please feel free to let us know.

Cheat Sheets

• Python Basics
• Pandas Basics
• Numpy Basics
• Scikit-Learn
• RDD Basics
• DataFrame Basics
• Apache Spark Cheat Sheet

Contents

• 01-data-strcture
  • 1.1-rdd.ipynb
  • 1.2-dataframe.ipynb
  • 1.3-conversion-between-rdd-and-dataframe.ipynb
  • 1.4-merge-and-split-columns.ipynb
• 02-data-manipulation
  • 2.1-map-functions.ipynb
  • 2.2-aggregate-functions.ipynb
  • 2.3-continuous-variable-to-categorical-variable.ipynb
  • 2.4-first-data-check.ipynb
  • 2.5-subset-dataframe-by-row.ipynb
  • 2.6-subset-dataframe-by-column.ipynb
  • 2.7.1-column-expression.ipynb
  • 2.7.2-dot-column-expression.ipynb
  • 2.7.3-boolean-column-expression.ipynb
  • 2.8-sql-functions-to-extend-column-expressions.ipynb
  • 2.9-user-defined-sql-function (udf).ipynb
  • import-and-export-data.ipynb
• 03-data-preparation
  • stringindexer-and-onehotencoder.ipynb
  • vector-assembler.ipynb
• 04-miscellaneous
  • add-python-files-to-spark-cluster.ipynb
  • dense-vs-sparse-vectors.ipynb
  • issues-and-solutions.ipynb
  • pipeline.ipynb
  • sql-functions.ipynb
  • TF-IDF.ipynb
• 05-module-turning
  • cross-validation.ipynb
  • regularization.ipynb
• 06-machine-learning
  • classification
    • binary-classification.ipynb
    • decision-tree-classification.ipynb
    • gradient-boost-tree-classification.ipynb
    • logistic-regression.ipynb
    • naive-bayes-classification.ipynb
    • random-forest-classification.ipynb
  • regression
    • generalized-linear-regression.ipynb
    • linear-regression.ipynb
• 07-natural-language-processing
  • nlp-and-nltk-basics.ipynb
  • nlp-information-extraction.ipynb
  • skills-needed-for-nlp-jobs.ipynb

Feedback and suggestions

Your comments and suggestions are highly appreciated. We are more than happy to receive corrections, suggestions or feedbacks for improvements.