Descriptive Analysis

Descriptive Analysis

            Descriptive analysis is one of the most fundamentals of any data. This is because it gives any reader the general idea of trends in the data provided. In the data provided about sale prices of properties in Manhattan from August 2012 to August 2013, the first observation made is that there is absolutely no data regarding easements. Therefore, this makes that specific column irrelevant. Also, the data provided in the borough section is not necessary as it is evident that all the properties herein are in the same borough. When it comes to the data provided under the tax class at present, the data range is tiny. There are just two categories of tax classes those are 2 and 4. The rest do not have data concerning the tax class. Considering or regarding the whole table of data, it is the tax class that has the smallest range of data entry.

There are some missing data in the building class in the present column. Out of 27400 entries, only 22 are missing; this results in a mere 0.08% of missing data. The addresses provided in the data are in the form of clusters in that several houses are living at a particular address. Picking few from the data, there are six houses at the 1 5TH AVENUE, 15E, one house at 1 5TH AVENUE, 15E, and seven houses at the 1 5TH AVENUE, 15E. There is quite a lot of data missing from the apartment column, as most houses don’t have their apartment number indicated. When it comes to zip codes of the houses, all data is present as all the zip code entries have been done..

The next set of data starting from residential units, commercial units, total units, square land feet, gross square feet, year built, and tax class at the time of sale have quite similar characteristics. Some of the traits include the fact that a lot of entries have not been entered or have a zero as an entry. Also, the data is not consistent, as there is a lot of it missing. There is also an observation that tax class at present has not changed; rather, it is not different from tax class at the time of sale. The two categories can be joined, or one of them struck off. Finally, data given under BUILDING CLASS AT TIME OF SALE is useful clean data as it is consistent, and all of it is there. Data for sale price and the sale date is inconsistent, and most of it was missing.

Data cleansing and Visualization

For one to come up with an efficient data cleansing practice, they first have to understand the goals and expectations for cleaning the data and then come up with a plan to execute the same. How are we going to maintain the hygiene of the data through a continuous process? The point of entry needs to be standardized because it is at this point where dirty data is absorbed. The other aspect is identifying duplicates because they essentially waste the effort to get clean data. Looking at the data we have, there are several duplicates. Some information is totally missing, and some are entirely irrelevant and thus should be cleaned before the same is used for prediction.

Model Building and Evaluation

Having done the descriptive analysis, it is evident that some features should be removed so s to improve the performance of the model.  Some of these features include low quality data, that which lacks representative categories and also those with noisy features. It is essential to identify nominal categorical features so that the metadata editor will have proper mathematical treatment. To make it simpler, it is advisable to clean all missing values by calculating the median and using it. To improve the accuracy of the results, all features should be treated differently rather than as a blanket transformation of columns. It is also important to note that not all the features that are there now will have a predictive value to the model. Some features may mislead the model or even add noise to it.

To use precise features, a Pearson correlation will be done to our data to test the features against the response class. From this process, only strong features will be picked to be used. The numbers or data that is obtained from correlation can also be fine-tuned to improve the performance of the model further. Under the algorithm, a gradient that can be regularized and a variant of the linear regression is to be used to reduce overfitting of the model.

There is a relationship in simple linear regression in that the magnitude of one variable. If one variable increases, let’s say, X, Y also increases, or it acan also be that when X increases, Y decreases. When with the correlation coefficient, the variable, which includes X and Y become interchangeable. When it comes to regression, what we are trying to achieve is predicting the Y variable from the X using a linear relationship, and that is using a line. Thus, we will end up with the equation

Y= b 0 + b1x

This equation will be read as, Y equals b₁ times X, and then plus a constant b₀. Two symbols have been used, and that is b and 0; they are interpreted as b 0 being the constant and then b 1 being the slope for X. Both of these symbols appear in the R outputs as coefficients, though the general use of the term coefficient has been reserved for b 1. In the equation, the Y variable, since it is dependent on X, it is referred to as a dependent variable. On the other side, the X variable is an independent variable. Other authors tend to refer to the X variable as a feature factor and the Y variable as the target. When there are multiple predictors in a data set, the equation will be extended in such a manner to accommodate them. Thus the equation will be,

Y= b 0 + b 1×1 + b 2×2 + …… + b p x p + e

As in our case, we will now have a linear model instead of a line because of the several predictors. For example, using our data, we will only use variables that we think are relevant and not all of the predictors. We will have such an equation;

House_imformula = sale price ~ building class at time of sale + land square feet + tax class + year built, = house, na.action = na.omit

            To measure the accuracy of the model, we will use the root mean squared error, which is the square root of the average error in y values. We will use the formula

RMSE = ∑i=1 n y I – y^ i 2 n