Week 8 Activity - Data preparation

Activity: Textual data pre-processing and informal analysis

Rule 1:

I created a list of positive words (unigrams and bigrams) from the given data and used them to identify positive and negative instances.

IF (effective OR intriguing OR breathtaking OR captivated OR (NOT not)_perfect OR loved OR real_chemistry OR really_good OR charm OR enthralled OR beautifully_done OR thoughtprovoking OR poignant OR fabulous OR sweet OR true_chemistry OR so_well OR enjoy OR excellent OR well_handled OR touching OR believable OR likeable OR very_successful OR enjoy OR interesting OR good OR entertaining OR great OR believable OR engaging) THEN pos ELSE neg

This rule doesn't apply correctly on all negative instances since some of them have positive words also. 

Rule 2:

This rule is based on a list of negative words from the given data

IF (not_perfect OR dull OR onedimensional OR misused OR unnatural OR lack OR missmarketed OR went_wrong OR worst OR shallow OR awful OR terrible OR really_bad OR cliché OR waste OR unintentional_laughs OR silliness OR immaturity OR passionless OR false_hope OR collapse OR annoying OR undercut OR not_so_well OR disaster OR not_original) THEN neg ELSE pos

This rule predicts some positive instances wrongly since a few of negative words occur in positive instances. 

Rule 3: 

To overcome the issue of wrong predictions due to some instances containing both positive and negative words, I used count to see which dominates which.

FOR ALL(effective OR intriguing OR breathtaking OR captivated OR (NOT not)_perfect OR loved OR real_chemistry OR really_good OR charm OR enthralled OR beautifully_done OR thoughtprovoking OR poignant OR fabulous OR sweet OR true_chemistry OR so_well OR enjoy OR excellent OR well_handled OR touching OR believable OR likeable OR very_successful OR enjoy OR interesting OR good OR entertaining OR great OR believable OR engaging) Add 1 to count_pos for each occurrence

FOR ALL(not_perfect OR dull OR onedimensional OR misused OR unnatural OR lack OR missmarketed OR went_wrong OR worst OR shallow OR awful OR terrible OR really_bad OR cliché OR waste OR unintentional_laughs OR silliness OR immaturity OR passionless OR false_hope OR collapse OR annoying OR undercut OR not_so_well OR disaster OR not_original) Add 1 to count_neg for each occurrence

IF count_pos> count_neg, THEN pos

ELSE neg

Rule 4: 

We can see that the list of words are hand-picked based on our sample data, so the above rule over-fits to our data. I removed words which may have different contexts in different occurrences and maintained only words that are predictive at all occurrences.

FOR ALL(effective OR breathtaking OR loved OR (real OR true_chemistry) OR really_good OR enthralled OR beautifully_done OR thoughtprovoking OR fabulous OR excellent OR well_handled OR very_successful) Add 1 to count_pos for each occurrence

FOR ALL(dull OR unnatural OR missmarketed OR went_wrong OR worst OR shallow OR awful OR terrible OR really_bad OR waste OR silliness OR annoying) Add 1 to count_neg for each occurrence

IF count_pos> count_neg, THEN pos

ELSE neg

Even though the above rule seems to fit okay, it may not be very predictive of instances which contain words other than the ones listed or which contain an opposite context of a word. They can be captured to some extent by complex rules involving the proximity of word occurrence. More features can be added and tested by cross-validation until we get a model with reasonable reliability. My take away is that it is not at all an easy task! :)

Week 6 Activity

In the activity for Week 6, we were asked to calculate different metrics for assessing models which were discussed in Ryan Baker's unit of Behavior Detection and Model Assessment. Two data sets, classifier-data-asgn2.csv and regressor-data-asgn2.csv were given. 

I used Excel for these calculations and for the last metric (A' or AUC), I downloaded a plugin called XLSTAT from http://www.xlstat.com/en/ since SPSS didnot give the correct answer. I will detail out the steps which I followed to complete this activity containing 11 questions. I urge you to save all the steps since you may need the answer of previous steps to continue the next steps. To better understand the steps I've described, refer the lecture videos :)

Q1) Using regressor-data-asgn2.csv, what is the Pearson correlation between data and predicted (model)? (Round to three significant digits; e.g. 0.24675 should be written as 0.247) (Hint: this is easy to compute in Excel)

 Use the excel function CORREL or PEARSON to calculate the Pearson correlation for the regressor model using the given two input arrays of data. Round the number you get, instead of truncating it to get the correct answer.

Q2) Using regressor-data-asgn2.csv, what is the RMSE between data and predicted (model)? (Round to three significant digits; e.g. 0.24675 should be written as 0.247) (Hint: this is easy to compute in Excel)

Calculate the residual (difference between actual data and predicted model) and use those values for the array in the below formula:

=SQRT(SUMSQ(A2:A1001)/COUNTA(A2:A1001))

Q3) Using regressor-data-asgn2.csv, what is the MAD between data and predicted (model)? (Round to three significant digits; e.g. 0.24675 should be written as 0.247) (Hint: this is easy to compute in Excel)

 Calculate the absolute values of the previous residual values in an array =ABS(RMSE!A2:A1001) and average them. 

Q4) Using classifier-data-asgn2.csv, what is the accuracy of the predicted (model)? Assume a threshold of 0.5. (Just give a rounded value rather than including the decimal; e.g. write 57.213% as 57) (Hint: this is easy to compute in Excel)

Compute the column of predicted model values with Y based on the given threshold of 0.5 (If >0.5, then Y). Compare it with the no of Ys in data to find the number of agreements. Calculate "= no. of agreements/ total count" for the accuracy.

Q5) Using classifier-data-asgn2.csv, how well would a detector perform, if it always picked the majority (most common) class? (Just give a rounded value rather than including the decimal; e.g. write 57.213% as 57) (Hint: this is easy to compute in Excel)

Calculate "= number of disagreements/total count". Use previous step values.

Q6) Is this detector’s performance better than chance, according to the accuracy and the frequency of the most common class?

Answer Yes/No based on the previous values you got.

Q7) What is this detector’s value for Cohen’s Kappa? Assume a threshold of 0.5. (Just round to the first two decimal places; e.g. write 0.74821 as 0.75).

I calculated the agreements between data and prediction model to form the confusion matrix of the number of True Negatives(TN), True Positives (TP), False Positives (FP), False Negatives (FN) and listed them as below from O5 to O8 and then used a formula:

00 (TN)

11 (TP)

01 (FP)

10 (FN)

=((O5+O6)-((((O6+O7)*(O6+O8))/SUM(O5:O8))+(((O5+O7)*(O5+O8))/SUM(O5:O8))))/((SUM(O5:O8))-((((O6+O7)*(O6+O8))/SUM(O5:O8))+(((O5+O7)*(O5+O8))/SUM(O5:O8))))

Alternatively, you may apply the values from your confusion matrix to any online calculator for Cohen's Kappa.

Q8) What is this detector’s precision, assuming we are trying to predict “Y” and assuming a threshold of 0.5 (Just round to the first two decimal places; e.g. write 0.74821 as 0.75).

Use formula Precision = TP/ (TP+FP)

Q9) What is this detector’s recall, assuming we are trying to predict “Y” and assuming a threshold of 0.5 (Just round to the first two decimal places; e.g. write 0.74821 as 0.75).

Use formula Recall = TP/ (TP+FN)

Q10) Based on the precision and recall, should this detector be used for strong interventions that have a high cost if mis-applied, or fail-soft interventions with low benefit and a low cost if mis-applied?

Select the correct option from the list of options.

Q11) What is this detector's value for A'? (Hint: There are some data points with the exact same detector confidence, so it is probably preferable to use a tool that computes A', such as http://www.columbia.edu/~rsb2162/computeAPrime.zip -- rather than a tool that computes the area under the ROC curve).

I used ROC Curve from XLSTAT plugin to compute Area under the curve (AUC) using excel. 

To compute A' without ROC curve, you may follow our co-learner's steps listed in his blog:

http://logort.com/2014/12/09/how-to-compute-auc-without-roc-curve/

Hope this helps you to reach this screen! :)

Competency 7.4

Competency 7.4: Describe how models might be used in Learning Analytics research, specifically for the problem of assessing some reasons for attrition along the way in MOOCs.

One particular model described by Dr. Carolyn talks about how certain properties of discussion correlate to dropout in MOOC. It explores how analyses of sentiment predict attrition over time (Sentiment however was found to be the least consistent and weakest indicator for dropouts). Refer the article below:

Survival Modeling:

Survival model is a regression model that captures the changes in probability of survival over time. It captures the probability at each time point and it is measured in terms of  hazard ratio which indicates how much more or less likely a student is to drop out. If Hazard ratio>1, the student is significantly more likely to drop out in the next time point.

Sentiment analysis in MOOC forums looked at Expressed sentiment and Exposure to sentiment. The four independent variables Individual Positivity, Individual Negativity, Thread Positivity and Thread Negativity were used to calculate the dependent variable Dropout. The effects were relatively weak and inconsistent across courses.

Some factors that may contribute to student attrition like student's prior motivation, skill set/ knowledge in the area, previous experience in learning MOOCs are difficult to capture. We can link different analysis methods like social network analysis, text mining, predictive modeling and survey data analysis to try to get the complete picture of an individual student for more consistent results. 

Competency 7.3/ Assignment

Building a simple text classification experiment - Training and evaluating a simple predictive model

I used LightSIDE tool as explained by Dr. Carolyn to run a simple classification experiment. The tool is easy to use and straightforward if we follow the steps. 

In the Extract Features pane, I loaded the NewsgroupTopic dataset from the sample data directory in LightSIDE. I selected Unigram and Bigram features and clicked on Extract. I then saved the feature space for later use.

In the Build Models pane, I used the recently created feature table. I selected Naive Bayes as the learning plugin, set the number of folds to be 20 for cross-validation and clicked on Train. 

I got Accuracy 58% and Kappa = 0.44 for the model as given in the assignment, which means my steps were correct :)

Competency 7.1/ 7.2 Text Mining

Text Mining is the process of extracting and identifying useful and meaningful information, from different sources of unstructured text data.

Prominent Areas of Text Mining

Information Retrieval:

Information Retrieval is the process of searching and retrieving the required document from a collection of documents based on the given search query. The search engines we use like Google, Yahoo etc. make use of IR techniques for matching and returning documents relevant to the user's query.

Document Classification/ Text Categorization:

Classification is the process of identifying the category a new observation belongs to, on the basis of a training set consisting of data with pre-defined categories (supervised learning). An example is the classification of email into spam/non-spam.

Clustering:

Clustering is the unsupervised procedure of classification where a set of similar objects are grouped to a cluster. An example analysis would be the summarization of common complaints based on open-ended survey responses.

Trend Analysis:

Trend Analysis is the process of discovering the trends of different topics over a given period of time. It is widely applied in summarizing news events and social network trends. An example would be the prediction of stock prices based on news articles.

Sentiment Analysis:

Sentiment analysis is the process of categorizing opinions based on sentiments like positive, negative or neutral. Sample applications include identifying sentiments in movie reviews and gaining real-time awareness to users' feedback.

Sub-area of Text Mining

Collaborative Learning Process Analysis

It is the process of analyzing the collaborative learning process of students using text mining techniques. Different indicators and language features are used for this study. Some of them are:

• General indicators of interactivity
• Turn length
• Conversation Length
• Number of student questions
• Student to tutor word ratio
• Student initiative
• Features related to cognitive processes
• Transactivity

Data familiarity in the domain is important to understand and develop features that are relevant.

Competency 6.2: Key Diagnostic Metrics

A part of Week 6 was designed to learn about the diagnostic metrics, to see how well our model does, as either classifiers or regressors. 

Metrics for Classifiers

Accuracy:

The easiest measure of model goodness is accuracy. It is also called agreement, when measuring the inter-rater reliability.

Accuracy = # of agreements/ Total # of assessments

It is generally not considered a good metric across fields, since it has non even assignment to categories and not useful. E.g. 92% accuracy in the Kindergarten Failure Detector Model in the extreme case always says Pass.

Kappa:

Kappa = (Agreement - Expected Agreement) / (1 - Expected Agreement)

If Kappa value
= 0, agreement is at chance
= 1, agreement is perfect
= negative infinity, agreement is perfectly inverse
> 1, something is wrong
< 0, agreement is worse than chance
0<Kappa<1, no absolute standard. For data-mined models, 0.3-0.5 is considered good enough for publishing.
Kappa is scaled by the proportion of each category, influenced by the data set. We can compare the Kappa values within the same data set, but not between two data sets.

ROC:

The Receiver Operating Characteristic Curve (ROC) is used while a model predicts something having two values (E.g correct/incorrect, dropout/not dropout) and outputs a probability or other real value (E.g. Student will drop out with 73% probability). 

It takes any number as cut-off (threshold) and some number of predictions (maybe 0) may then be classified as 1's and the rest may be classified as 0s. There are four possibilities for a classification threshold:
True Positive (TP) - Model and the Data say 1
False Positive (FP) - Data says 0, Model says 1
True Negative (TN) - Model and the Data say 0
False Negative (FN) - Data says 1, Model says 0

The ROC Curve has in its X axis Percent False Positives (Vs. True Negatives) and in Y axis Percent True Positives (Vs. False Negatives). The model is good if it is above the chance line in its diagonal.

A':

A' is the probability that if the model is given an example from each category, it will accurately identify which is which. It is a close relative of ROC and mathematically equivalent to Wilcoxon statistic. It gives useful result, since we can compute statistical tests for:
- whether two A' values are significantly different in the same or different data sets.
- whether an A' value is significantly different than choice.

A' Vs Kappa:

A' is more difficult to compute and works only for 2 categories. It's meaning is invariant across data sets i.e) A'=0.6 is always better than A'=0.5. It is easy to interpret statistically and has value almost always higher than Kappa values. It also takes confidence into account.

Precision and Recall:

Precision is the probability that a data point classified as true is actually true.
Precision = TP / (TP+FP)
Recall is the probability that a data point that is actually true is classified as true.
Recall = TP / (TP+FN)
They don't take confidence into account.

Metrics for Regressors

Linear Correlation (Pearson correlation):

In r(A,B) when A's value changes, does B change in the same direction?
It assumes a linear relationship.
If correlation value is
1.0 : perfect
0.0 : none
-1.0 : perfectly negatively correlated
In between 0 and 1 : Depends on the field
0.3 is good enough in education since a lot of factors contribute to just any dependent measure.
Different functions (outliers) may also have the same correlation.

R square:

R square is correlation squared. It is the measure of what percentage of variance in dependent dependent measure is explained by a model. If predicting A with B,C,D,E, it is often used as the measure of model goodness rather than r.

MAE/MAD:

Mean Absolute Error/ Deviation is the average of absolute value of actual value minus predicted value. i.e) the average of each data point's difference between actual and predicted value. It tells the average amount to which the predictions deviate from the actual value and is very interpret able.

RMSE:

Root Mean Square Error (RMSE) is the square root of average of (actual value minus predicted value)^2. It can be interpreted similar to MAD but it penalizes large deviation more than small deviation. It is largely preferred to MAD. Low RMSE is good.

RMSE/ MAD	Correlation	Model
Low	High	Good
High	Low	Bad
High	High	Goes in the right direction, but systematically biased
Low	Low	Values are in the right range, but doesn’t capture relative change

Information Criteria:

BiC:

Bayesian Information Criterion (BiC) makes trade-off between goodness of fit and flexibility of fit (number of parameters). The formula for linear regression:

BiC' = n log (1-r^2) + p log n 

where n - number of students, p - number of variables

If value > 0, worse than expected, given number of variables

   value <0, better than expected, given number of variables

It can be used to understand the significance of difference between models. (E.g. 6 implies statistically significant difference)

AiC:

An Information Criterion/ Akaike's Information Criterion (AiC) is an alternative to BiC. It has slightly different trade-off between goodness and flexibility of fit.

Note: There is no single measure to choose between classifiers. We have to understand multiple dimensions and use multiple metrics.

Types of Validity

Generalizability:

Does your model remain predictive when used in a new data set?
Generalizability underlies the cross-validation paradigm that is common in data mining. Knowing the context of the model where it will be used in, drives the kind of generalization to be studied.
Fail: Model of boredom built on data from 3 students fails when applied to new students

Ecological Validity:

Do your findings apply to real-life situations outside of research settings?
E.g. If a behavior detector built in lab settings work in real classrooms.

Construct Validity:

Does your model actually measure what it was intended to measure?
Does your model fir the training data? (provided the training data is correct)

Predictive Validity:

Does your model predict not just the present, but the future as well?

Substantive Validity:

Does your results matter?

Content Validity:

From testing; Does your test cover the full domain it is meant to cover?
For behavior modeling, does the model cover the full range of behavior it is intended to?

Conclusion Validity:

Are your conclusions justified based on evidence?

I think that the lessons in Week 5 and 6 are very useful, especially when we want to get our hands deep into predictive modeling and diagnosing its usefulness. I hope to use them in my predictive modeling work :)

Competency 6.1: Engineer both feature and training labels

My notes/ learning

Behavior Detectors

Behavior detectors are automated (predictive) models that can infer from log files whether a student is behaving in a certain way. 

Behaviors:

Disengaged behaviors 

-gaming the system by trying to succeed without learning

-off-task behavior

-carelessness by giving wrong answer even when having the required skills

-WTF behavior - Without Thinking Fastidiously (by doing unrelated tasks while using the system)

Metacognitive behaviors

-help-avoidance

-unscaffolded self-exploration

-exploration behaviors

Related Problem:

-sensor-free affect detection (without the use of video-capture, gesture capture etc.)

- detecting boredom, frustration, engaged concentration, delight

Ground Truth

Ground truth is the accuracy of classification in supervised learning/ machine learning.

Where to get the prediction labels from is the big issue in developing behavior detectors.

E.g. How to identify when a student is off-task/ gaming the system?

Behavior labels are noisy; there is no perfect way to get indicators of student behavior.

Sources of ground truth:

• Self-Report: -common for affect, self-efficacy; not common for labeling behavior (students may not admit gaming)
•  Field observations
• Text replays
• Video coding

Field observations:

One or more observers watch students and take notes

- requires training to do it right

Text Replays:

Analyzing student interaction behavior from log files based on their input in the system.

- Fast to conduct

- Decent inter-rater reliability

- Agrees with other measures of constructs

- Can be used to train behavior detectors

- Only limited constructs can be coded

- Lower precision than field observation due to lower bandwidth

Video Coding:

Videos of live behavior in the classrooms or screen replay videos analyzed

- slowest, but replicable and precise

- challenges in camera positioning

Kappa= 0.6 or higher expected for expert coding

However, 1000 data points with kappa= 0.5 > 100 data points with kappa= 0.7

Once we have ground truth, we can build the detector.

Feature Engineering

Feature engineering is the art of creating predictor variables. The model will not be good if our features (predictors) are not good. It involves lore rather than well-known and validated principles.

The big idea is how we can take the voluminous, ill-formed and yet under-specified data that we now have in education and shape it into a reasonable set of variables in an efficient and predictive way.

Process:

1. Brainstorming features - IDEO tips for brainstorming
2. Deciding what features to create - trade-off between effort and usefulness of feature
3. Creating the features - Excel, OpenRefine, Distillation code
4. Studying the impact of features on model goodness
5. Iterating on features if useful - try close variants and test
6. Go to 3 (or 1)

Feature engineering can over-fit --> Iterate and use cross-validation, test on held-out data or newly collected data.

Thinking about our variables is likely to yield better results than using pre-existing variables from a standard set.

Knowledge Engineering and Data Mining:

Knowledge engineering is where the model is created by a smart human being, rather than an exhaustive computer (that searches through all possibilities). It is also called rational modeling or cognitive modeling.

At its best:

Knowledge engineering is the art of a human being becoming deeply familiar with the target construct by carefully studying the data, including possible process data, understanding the relevant theory and thoughtfully crafting an excellent model.

-achieves higher construct validity and comparable performance than data mining

-may even transfer better to new data (while data-mined model may get trapped at finding specific features to the population)

E.g. Alevan et al.'s (2004, 2006) Help-seeking model

It was developed based on scientific articles, experience in designing learning environments, log files of student interaction and experience watching students using educational software in classes.

At its worst:

If it refers to making up a simple model very quickly and calling the resultant construct by a well-known name, not testing on data or has no evidence.

- poorer construct validity than data mining

- predicts desired constructs poorly 

- can slow scientific progress by false results
- can hurt student outcomes by wrong intervention

It is easier to identify if a data mining model is bad, from the features, validation procedure or goodness metrics; but difficult for knowledge engineering since the hard-work process in researcher's brain is invisible.

To Do's for both methods:

- Test the models

- Use direct measures (Training labels) or Indirect measures (E.g. predicting student learning). 

- Careful study of construct leads to better features and better models

Assignment - Critical Reflection:

Possible uses in education:

Behavior detection can be used to create automated learning management systems that will give hints to users/ comment on their performances by detecting their behavior. It can be used in places where the tutor is not available to help all students. The online automated tutor can jump in to give suggestions. If the behavior is still detected to be disengaged, an available tutor can be mapped to the student.