You may consult with other students currently taking CAP 5625 in your section at FAU on this programming assignment. If you do consult with others, then you must indicate this by providing their names with your submitted assignment. However, all analyses must be performed independently, all source code must be written independently, and all students must turn in their own independent assignment. Note that for this assignment, you may choose to pair up with one other student in your section of CAP 5625 and submit a joint assignment. If you choose to do this, then both your names must be associated with the assignment and you will each receive the same grade.
Though it should be unnecessary to state in a graduate class, I am reminding you that you may not turn in code (partial or complete) that is written or inspired by others, including code from other students, websites, past code that I release from prior assignments in this class or from past semesters in other classes I teach, or any other source that would constitute an academic integrity violation. All instances of academic integrity violations will receive a zero on the assignment and will be referred to the Department Chair and College Dean for further administrative action. A second offense could lead to dismissal from the University and any offense could result in ineligibility for Departmental Teaching Assistant and Research Assistant positions.
You may choose to use whatever programming language you want. However, you must provide clear instructions on how to compile and/or run your source code. I recommend using a modern language, such as Python, R, or Matlab as learning these languages can help you if you were to enter the machine learning or artificial intelligence field in the future. All analyses performed and algorithms run must be written from scratch. That is, you may not use a library that can perform gradient descent, cross validation, ridge regression, logistic (multinomial) regression, optimization, etc. to successfully complete this assignment (though you may reuse your relevant code from Programming Assignments 1, 2, and 3). The goal of this assignment is not to learn how to use particular libraries of a language, but it is to instead understand how key methods in statistical machine learning are implemented. With that stated, I will provide 5% extra credit if you additionally implement the assignment using built-in statistical or machine learning libraries (see Deliverable 7 at end of the document). Note, credit for deliverables that request graphs, discussion of results, or specific values will not be given if the instructor must run your code to obtain these graphs, results, or specific values. Brief overview of assignment
Mobile User
2
In this assignment you will still be analyzing human genetic data from ๐ = 183 training observations (individuals) sampled across the world. The goal is to fit a model that can predict (classify) an individualโs ancestry from their genetic data that has been projected along ๐ = 10 top principal components (proportion of variance explained is 0.2416) that we use as features rather than the raw genetic data, as the training would take too long to complete for this assignment with the raw data. I recognize that we have not covered precisely what principal components are, but we will get to this in Module 10, and for now you should just treat them as highly informative features as input to your classifier. Specifically, you will perform a penalized (regularized) logistic (multinomial) regression fit using ridge regression, with the model parameters obtained by batch gradient descent. Your predictions will be based on ๐พ = 5 continental ancestries (African, European, East Asian, Oceanian, or Native American). Ridge regression will permit you to provide parameter shrinkage (tuning parameter ๐ โฅ 0) to mitigate overfitting. The tuning parameter ๐ will be chosen using five-fold cross validation, and the best- fit model parameters will be inferred on the training dataset conditional on an optimal tuning parameter. This trained model will be used to make predictions on new test data points. Training data
Training data for these observations are given in the attached TrainingData_N183_p10.csv comma-separated file, with individuals labeled on each
row (rows 2 through 184), and input features (PC1, PC2, โฆ, PC10) and ancestry label given on the columns (with the first row representing a header for each column). Test data
Test data are given in the attached TestData_N111_p10.csv comma-separated file, with individuals labeled on each row (rows 2 through 112), and input features (PC1, PC2, โฆ, PC10), and ancestry label given on the columns (with the first row representing a header for each column). There are five individuals with Unknown ancestry, 54 individuals with Mexican ancestry, and 52 individuals with African American ancestry. Each of the five Unknown individuals belong to one of the five ancestries represented in the training set, and each ancestry is represented once in the five Unknown individuals. The Mexican and African American individuals have a range of ancestry proportions based on historical mixing of ancestors of diverse ancestry. You will use the class probabilities from logistic (multinomial) regression to predict the ancestry proportion of each of these putatively mixed samples. Detailed description of the task
Recall that the task of performing a ridge-penalized logistic regression fit to training data {(๐ฅ1, ๐ฆ1), (๐ฅ2, ๐ฆ2), โฆ , (๐ฅ๐ , ๐ฆ๐)} is to minimize the cost function
๐ฝ(๐, ๐) = โ log โ(๐) + ๐โโ๐ฝ๐๐ 2
๐
๐=1
๐พ
๐=1
where
3
๐ = [
๐ฝ01 ๐ฝ02 โฏ ๐ฝ0๐พ ๐ฝ11 ๐ฝ12 โฏ ๐ฝ1๐พ โฎ โฎ โฑ โฎ ๐ฝ๐1 ๐ฝ๐2 โฏ ๐ฝ๐๐พ
]
is the (๐ + 1) ร ๐พ matrix of model parameters, where
log โ(๐) = โ[โ๐ฆ๐๐ (๐ฝ0๐ +โ๐ฅ๐๐๐ฝ๐๐
๐
๐=1
)
๐พ
๐=1
โ log(โexp(๐ฝ0โ +โ๐ฅ๐๐๐ฝ๐โ
๐
๐=1
)
๐พ
โ=1
)]
๐
๐=1
is the likelihood function, where ๐ฆ๐๐, ๐ = 1,2, โฆ ,๐ and ๐ = 1,2, โฆ , ๐พ, is an indicator variable defined as
๐ฆ๐๐ = { 1 if ๐ฆ๐ = ๐ 0 if ๐ฆ๐ โ ๐
and where the input ๐ features are standardized (i.e., centered and divided by their standard deviation). Moreover, recall that batch gradient descent updates each parameter ๐ฝ๐๐, ๐ =
0,1, โฆ , ๐ and ๐ = 1,2, โฆ , ๐พ, as follows:
๐ฝ๐๐ โ ๐ฝ๐๐ โ ๐ผ ๐
๐๐ฝ๐๐ ๐ฝ(๐, ๐)
where ๐ผ is the learning rate and where the partial derivative of the cost function with respect to parameter ๐ฝ๐๐ is
๐
๐๐ฝ๐๐ ๐ฝ(๐, ๐) =
{
โโ[๐ฆ๐๐ โ ๐๐(๐ฅ๐; ๐)]
๐
๐=1
if ๐ = 0
โโ๐ฅ๐๐[๐ฆ๐๐ โ ๐๐(๐ฅ๐; ๐)]
๐
๐=1
+ 2๐๐ฝ๐๐ if ๐ > 0
where
๐๐(๐ฅ๐; ๐) = exp(๐ฝ0๐ + โ ๐ฅ๐๐๐ฝ๐๐
๐ ๐=1 )
โ exp (๐ฝ0โ + โ ๐ฅ๐๐๐ฝ๐โ ๐ ๐=1 )๐พ
โ=1
To implement this algorithm, depending on whether your chosen language can quickly compute vectorized operations, you may implement batch gradient descent using either Algorithm 1 or Algorithm 2 below (choose whichever you are more comfortable implementing). Note that in languages like R, Python, or Matlab, Algorithm 2 (which would be implemented by several nested loops) may be much slower than Algorithm 1. Also note that if you are implementing Algorithm 1 using Python, use numpy arrays instead of Pandas data frames for computational speed. For this assignment, assume that we will reach the minimum of the cost function within a fixed number of steps, with the number of iterations being 10,000.
4
You may need to explore different learning rate values to identify one that is not too large and not too small, such that it is likely for the algorithm to converge in a reasonable period of time. I would consider a learning rate of ๐ผ = 10โ5, though I encourage you to explore how your model trains for smaller and larger learning rates as well. For this assignment, assume that we will reach the minimum of the cost function within a fixed number of steps, with the number of iterations being 10,000 (a large number as we have many parameters). Keep in mind that due to this large number of iterations, it could take a long time to train your classifier.
Algorithm 1 (vectorized): Step 1. Choose learning rate ๐ผ and fix tuning parameter ๐ Step 2. Generate ๐ ร (๐ + 1) augmented design matrix
๐ =
[ 1 ๐ฅ11 ๐ฅ12 โฏ ๐ฅ1๐ 1 ๐ฅ21 ๐ฅ22 โฏ ๐ฅ2๐ โฎ โฎ โฎ โฑ โฎ 1 ๐ฅ๐1 ๐ฅ๐2 โฏ ๐ฅ๐๐]
where column ๐ + 1 has been centered and standardized such that feature ๐, ๐ = 1,2, โฆ , ๐, has mean zero and standard deviation one, and generate ๐ ร ๐พ indicator response matrix
๐ = [
๐ฆ11 ๐ฆ12 โฏ ๐ฆ1๐พ ๐ฆ21 ๐ฆ22 โฏ ๐ฆ2๐พ โฎ โฎ โฑ โฎ ๐ฆ๐1 ๐ฆ๐2 โฏ ๐ฆ๐๐พ
]
where ๐ฆ๐๐ = 1 if observation ๐ is from class ๐, and 0 otherwise. Step 3. Initialize the (๐ + 1) ร ๐พ-dimensional parameter matrix
๐ = [
๐ฝ01 ๐ฝ02 โฏ ๐ฝ0๐พ ๐ฝ11 ๐ฝ12 โฏ ๐ฝ1๐พ โฎ โฎ โฑ โฎ ๐ฝ๐1 ๐ฝ๐2 โฏ ๐ฝ๐๐พ
]
to all zeros, so that initially each class has the same probability. Step 4. Compute ๐ ร ๐พ unnormalized class probability matrix as
๐ = [
๐ข11 ๐ข12 โฏ ๐ข1๐พ ๐ข21 ๐ข22 โฏ ๐ข2๐พ โฎ โฎ โฑ โฎ ๐ข๐1 ๐ข๐2 โฏ ๐ข๐๐พ
] = exp(๐๐)
where exp(๐๐) indicates exponentiation of each element of the ๐๐ matrix, and not the matrix exponential of ๐๐.
Step 5. Compute ๐ ร ๐พ normalized class probability matrix as
๐ = [
๐11 ๐12 โฏ ๐1๐พ ๐21 ๐22 โฏ ๐2๐พ โฎ โฎ โฑ โฎ ๐๐1 ๐๐2 โฏ ๐๐๐พ
]
5
where
๐๐(๐ฅ๐; ๐) = ๐๐๐ = ๐ข๐๐
โ ๐ข๐โ ๐พ โ=1
Step 6. For ease of vectorization, generate (๐ + 1) ร ๐พ intercept matrix
๐ = [
๐ฝ01 ๐ฝ02 โฏ ๐ฝ0๐พ 0 0 โฏ 0 โฎ โฎ โฑ โฎ 0 0 โฏ 0
]
Step 7. Update the parameter matrix as
๐ โถ= ๐ + ๐ผ[๐๐(๐ โ ๐) โ 2๐(๐ โ ๐)]
Step 8. Repeat Steps 4 to 7 for 10,000 iterations
Step 9. Set the last updated parameter matrix as ๏ฟฝฬ๏ฟฝ
Algorithm 2 (non-vectorized): Step 1. Choose learning rate ๐ผ and fix tuning parameter ๐ Step 2. Generate ๐ ร ๐ design matrix
๐ = [
๐ฅ11 ๐ฅ12 โฏ ๐ฅ1๐ ๐ฅ21 ๐ฅ22 โฏ ๐ฅ2๐ โฎ โฎ โฑ โฎ ๐ฅ๐1 ๐ฅ๐2 โฏ ๐ฅ๐๐
]
where column ๐ has been centered and standardized such that feature ๐, ๐ = 1,2, โฆ , ๐, has mean zero and standard deviation one, and generate ๐ ร ๐พ indicator response matrix
๐ = [
๐ฆ11 ๐ฆ12 โฏ ๐ฆ1๐พ ๐ฆ21 ๐ฆ22 โฏ ๐ฆ2๐พ โฎ โฎ โฑ โฎ ๐ฆ๐1 ๐ฆ๐2 โฏ ๐ฆ๐๐พ
]
where ๐ฆ๐๐ = 1 if observation ๐ is from class ๐, and 0 otherwise. Step 3. Initialize the (๐ + 1) ร ๐พ-dimensional parameter matrix
๐ = [
๐ฝ01 ๐ฝ02 โฏ ๐ฝ0๐พ ๐ฝ11 ๐ฝ12 โฏ ๐ฝ1๐พ โฎ โฎ โฑ โฎ ๐ฝ๐1 ๐ฝ๐2 โฏ ๐ฝ๐๐พ
]
to all zeros, so that initially each class has the same probability. Step 4. Create temporary (๐ + 1) ร ๐พ-dimensional parameter matrix
6
๐temp =
[ ๐ฝ01
temp ๐ฝ02 temp
โฏ ๐ฝ0๐พ temp
๐ฝ11 temp
๐ฝ12 temp
โฏ ๐ฝ1๐พ temp
โฎ โฎ โฑ โฎ
๐ฝ๐1 temp
๐ฝ๐2 temp
โฏ ๐ฝ๐๐พ temp
]
Step 5. Compute ๐ ร ๐พ unnormalized class probability matrix as
๐ = [
๐ข11 ๐ข12 โฏ ๐ข1๐พ ๐ข21 ๐ข22 โฏ ๐ข2๐พ โฎ โฎ โฑ โฎ ๐ข๐1 ๐ข๐2 โฏ ๐ข๐๐พ
]
where
๐ข๐๐ = exp(๐ฝ0๐ +โ๐ฅ๐๐๐ฝ๐๐
๐
๐=1
)
Step 6. Compute ๐ ร ๐พ normalized class probability matrix as
๐ = [
๐11 ๐12 โฏ ๐1๐พ ๐21 ๐22 โฏ ๐2๐พ โฎ โฎ โฑ โฎ ๐๐1 ๐๐2 โฏ ๐๐๐พ
]
where
๐๐(๐ฅ๐; ๐) = ๐๐๐ = ๐ข๐๐
โ ๐ข๐โ ๐พ โ=1
Step 7. For each ๐, ๐ = 0,1, โฆ , ๐, and ๐, ๐ = 1,2, โฆ , ๐พ, find next value for parameter ๐ for class ๐ as
๐ฝ๐๐ temp
โ
{
๐ฝ๐๐ + ๐ผ (โ[๐ฆ๐๐ โ ๐๐๐]
๐
๐=1
) if ๐ = 0
๐ฝ๐๐ + ๐ผ(โ๐ฅ๐๐[๐ฆ๐๐ โ ๐๐๐]
๐
๐=1
โ 2๐๐ฝ๐๐) if ๐ > 0
Step 8. Update the parameter matrix as ๐ = ๐temp
Step 9. Repeat Steps 5 to 8 for 10,000 iterations
Step 10. Set the last updated parameter matrix as ๏ฟฝฬ๏ฟฝ Effect of tuning parameter on inferred regression coefficients You will consider a discrete grid of nine tuning parameter values ๐ โ {10โ4, 10โ3, 10โ2, 10โ1, 100, 101, 102, 103, 104} where the tuning parameter is evaluated across a wide range of values on a log scale. For each tuning parameter value, you will use batch gradient descent to infer the best-fit model.
7
Deliverable 1: Illustrate the effect of the tuning parameter on the inferred ridge regression coefficients by generating five plots (one for each of the ๐พ = 5 ancestry classes) of 10 lines
(one for each of the ๐ = 10 features), with the ๐ฆ-axis as ๏ฟฝฬ๏ฟฝ๐๐, ๐ = 1,2, โฆ ,10 for the graph of
class ๐, and ๐ฅ-axis the corresponding log-scaled tuning parameter value log10(๐) that
generated the particular ๏ฟฝฬ๏ฟฝ๐๐. Label both axes in all five plots. Without the log scaling of the
tuning parameter, the plot will look distorted. Choosing the best tuning parameter You will consider a discrete grid of nine tuning parameter values ๐ โ {10โ4, 10โ3, 10โ2, 10โ1, 100, 101, 102, 103, 104} where the tuning parameter is evaluated across a wide range of values on a log scale. For each tuning parameter value, perform five-fold cross validation and choose the ๐ value that gives the smallest
CV(5) = 1
5 โ CategoricalCrossEntropy๐
5
๐=1
where
CategoricalCrossEntropy๐ = โ 1
๐๐ โ (โ๐ฆ๐๐ log10 ๐๐(๐ฅ๐; ๏ฟฝฬ๏ฟฝ)
๐พ
๐=1
)
๐โ Validation Set ๐
is a measure of the cost for a classifier with ๐พ classes in fold ๐ and where ๐๐ is the number of observations in Validation set ๐. Note that during the five-fold cross validation, you will hold out one of the five sets (here either 36 or 37 observations) as the Validation Set and the remaining four sets (the other 147 or 146 observations) will be used as the Training Set. On this Training Set, you will need to standardize (center and divided by the standard deviation across samples) each feature. These identical values used for standardizing the input will need to be applied to the corresponding Validation Set, so that the Validation set is on the same scale. Because the Training Set changes based on which set is held out for validation, each of the five pairs of Training and Validation Sets will have different standardization parameters.
Deliverable 2: Illustrate the effect of the tuning parameter on the cross validation error by generating a plot with the ๐ฆ-axis as CV(5) error, and the ๐ฅ-axis the corresponding log-scaled
tuning parameter value log10(๐) that generated the particular CV(5) error. Label both axes
in the plot. Without the log scaling of the tuning parameter ๐, the plots will look distorted.
Deliverable 3: Indicate the value of ๐ value that generated the smallest CV(5) error.
Deliverable 4: Given the optimal ๐, retrain your model on the entire dataset of ๐ = 183
observations to obtain an estimate of the (๐ + 1) ร ๐พ model parameter matrix as ๏ฟฝฬ๏ฟฝ and
8
make predictions of the probability for each of the ๐พ = 5 classes for the 111 test individuals located in TestData_N111_p10.csv. That is, for class ๐, compute
๐๐(๐; ๏ฟฝฬ๏ฟฝ) = exp(๏ฟฝฬ๏ฟฝ0๐ + โ ๐๐๏ฟฝฬ๏ฟฝ๐๐
๐ ๐=1 )
โ exp (๏ฟฝฬ๏ฟฝ0โ + โ ๐๐๏ฟฝฬ๏ฟฝ๐โ ๐ ๐=1 )๐พ
โ=1
for each of the 111 test samples ๐, and also predict the most probable ancestry label as
๏ฟฝฬ๏ฟฝ(๐) = arg max ๐โ{1,2,โฆ,๐พ}
๐๐(๐; ๏ฟฝฬ๏ฟฝ)
Report all six values (probability for each of the ๐พ = 5 classes and the most probable ancestry label) for all 111 test individuals.
Deliverable 5: How do the class label probabilities differ for the Mexican and African American samples when compared to the class label probabilities for the unknown samples? Are these class probabilities telling us something about recent history? Explain why these class probabilities are reasonable with respect to knowledge of recent history.
Deliverable 6: Provide all your source code that you wrote from scratch to perform all analyses (aside from plotting scripts, which you do not need to turn in) in this assignment, along with instructions on how to compile and run your code. Deliverable 7 (extra credit): Implement the assignment using statistical or machine learning libraries in a language of your choice. Compare the results with those obtained above, and provide a discussion as to why you believe your results are different if you found them to be different. This is worth up to 5% additional credit, which would allow you to get up to 105% out of 100 for this assignment.
BIDS(63)
Select your paper details and see how much our professional writing services will cost.
Our custom human-written papers from top essay writers are always free from plagiarism.
Your data and payment info stay secured every time you get our help from an essay writer.
Your money is safe with us. If your plans change, you can get it sent back to your card.
We offer more than just hand-crafted papers customized for you. Here are more of our greatest perks.
Get instant answers to the questions that students ask most often.
See full FAQWe complete each paper from scratch, and in order to make you feel safe regarding its authenticity, we check our content for plagiarism before its delivery. To do that, we use our in-house software, which can find not only copy-pasted fragments, but even paraphrased pieces of text. Unlike popular plagiarism-detection systems, which are used by most universities (e.g. Turnitin.com), we do not report to any public databasesโtherefore, such checking is safe.
We provide a plagiarism-free guarantee that ensures your paper is always checked for its uniqueness. Please note that it is possible for a writing company to guarantee an absence of plagiarism against open Internet sources and a number of certain databases, but there is no technology (except forย turnitin.comย itself) that could guarantee no plagiarism against all sources that are indexed by turnitin. If you want to be 100% sure of your paperโs originality, we suggest you check it using the WriteCheck service fromย turnitin.comย and send us the report.
Yes. You can have a free revision during 7 days after youโve approved the paper. To apply for a free revision, please press the revision request button on your personal order page. You can also apply for another writer to make a revision of your paper, but in such a case, we can ask you for an additional 12 hours, as we might need some time to find another writer to work on your order.
After the 7-day period, free revisions become unavailable, and we will be able to propose only the paid option of a minor or major revision of your paper. These options are mentioned on your personal order page.