This page illustrates Cubist models and their predictive performance on some diverse applications. Like See5/C5.0, Cubist pays particular attention to the issue of comprehensibility. RuleQuest believes that a data mining system should find patterns that not only facilitate accurate predictions, but also provide insight. We hope this is evident in the following examples, all of which were run with Cubist's default parameter values. Times are for a 2.6GHz Core i7 running 64-bit Linux.
This first application uses data on housing prices circa 1980 in Boston tracts. Each case describes average characteristics of houses in a tract that might be expected to affect their price. Here are a few examples:
Abbrev Attribute Tract 1 Tract 2 Tract 3 ..... CRIM crime rate 7.67 2.24 0.08 ZN proportion large lots - - 45 INDUS proportion industrial 18.1 19.6 3.4 NOX nitric oxides ppm .69 .61 .44 RM av rooms per dwelling 5.7 5.9 7.2 AGE proportion pre-1940 98.9 91.8 38.9 DIS distance to employment centers 1.6 2.4 4.6 RAD accessibility to radial highways 24 5 5 TAX property tax rate per $10,000 666 403 398 PTRATIO pupil-teacher ratio 20.2 14.7 15.2 LSTAT percentage low income earners 19.9 11.6 5.4 PRICE average price ($'000) 8.5 22.7 36.4
From 506 cases like this, Cubist takes less than a tenth of a second to construct a model consisting of four rules:
Rule 1: [101 cases, mean 13.79, range 5 to 27.5, est err 2.21]
if
NOX > 0.668
then
CMEDV = 2.05 + 2.03 DIS - 0.37 LSTAT + 21.4 NOX - 0.06 CRIM
Rule 2: [203 cases, mean 19.42, range 7 to 31, est err 2.13]
if
NOX <= 0.668
LSTAT > 9.59
then
CMEDV = 31.13 + 2.5 RM - 0.24 LSTAT - 0.79 PTRATIO - 0.72 DIS
- 0.038 AGE - 3.6 NOX - 0.0024 TAX + 0.04 RAD - 0.03 CRIM
+ 0.007 ZN
Rule 3: [43 cases, mean 24.16, range 18.2 to 50, est err 2.68]
if
RM <= 6.226
LSTAT <= 9.59
then
CMEDV = -23.77 + 0.95 CRIM + 0.81 RAD + 8.5 RM - 0.83 LSTAT + 0.0075 TAX
- 0.4 DIS - 0.12 PTRATIO - 0.009 AGE + 0.005 ZN
Rule 4: [163 cases, mean 31.43, range 16.5 to 50, est err 2.76]
if
RM > 6.226
LSTAT <= 9.59
then
CMEDV = -4.79 + 2.27 CRIM + 9.2 RM - 0.83 LSTAT - 0.019 TAX - 0.72 DIS
- 0.7 PTRATIO - 0.039 AGE + 0.03 RAD - 1.8 NOX + 0.008 ZN
Each rule has three parts: some descriptive information, conditions that must be satisfied before the rule can be used, and a linear model.
How can a model like this be used to make predictions about new cases? With some slight suppression of details, the procedure is as follows:
All very well, you might say, but how good is the model? Here are the results of a 10-fold cross-validation on this dataset, plotting the predicted value of each unseen case against its real value.
Even though the model is quite simple, these results compare favorably with most published results for this dataset.
Statlib is a central repository used by statisticians. One of the datasets obtainable from this interesting site concerns estimating the fat content of meat samples using absorbency in the near infrared spectrum. This data comes from the Tecator Infratec Food and Feed Analyzer using 100 channels. Each attribute consists of the value of the instrument reading in one channel, so this is a high-dimensional prediction task.
Cubist derives a model with four rules from 240 training examples (again in less than 0.1 seconds):
Rule 1: [13 cases, mean 7.577, range 1.7 to 15.9, est err 0.933]
if
A05 > 3.17348
A89 <= 3.82829
then
Fat = -7.32 - 3037 A38 + 2932.7 A37 - 3411.5 A13 + 2798.8 A53
+ 2372.8 A39 + 2885.3 A12 - 2183.8 A54 - 1856.9 A34 - 1496.3 A98
- 1453.4 A40 - 1859.6 A05 + 1229.8 A99 + 1146.5 A57 + 980.3 A60
- 996.9 A52 - 949.7 A58 + 1090.4 A09 + 930.8 A30 + 801.6 A44
- 857.7 A28 - 723.3 A61 - 623.2 A95 + 477.5 A97 + 495.5 A25
+ 560 A07 - 445.4 A48 + 547.1 A00 + 416.4 A36 - 330.8 A45
+ 326.5 A49 + 303.4 A93 + 256.8 A90 - 176.6 A50 - 112.5 A89
- 32.8 A81
Rule 2: [129 cases, mean 11.667, range 0.9 to 36.2, est err 0.833]
if
A40 <= 3.08971
then
Fat = 7.263 + 6060 A38 - 5593.9 A37 + 5348.6 A36 + 4581.8 A53 + 5121 A12
- 3990.6 A40 - 3798.5 A34 - 4133.9 A05 - 2824.2 A95 - 2866.6 A52
- 3267.1 A17 + 2744.8 A60 + 2497.6 A97 - 2793.3 A13 - 2189.4 A58
- 2056.2 A54 - 2003.8 A70 + 2424 A07 - 1877 A30 + 1722.9 A39
+ 1705 A76 + 1582.1 A28 + 1595.4 A25 - 1210.7 A98 + 1079.5 A57
+ 942.4 A99 + 1026.7 A09 - 687.9 A61 + 635.7 A44 + 515.1 A00
- 350.3 A45 + 241.7 A90 - 179.7 A49
Rule 3: [35 cases, mean 26.331, range 10 to 56.6, est err 1.601]
if
A05 > 3.17348
A89 > 3.82829
then
Fat = 14.568 + 6740.3 A39 + 6256.3 A49 - 5734 A48 - 5374.5 A38
- 5371.3 A34 - 4599.1 A40 + 4257.9 A36 - 3282.7 A95 + 3508 A30
+ 3102.5 A93 - 2987 A98 + 2990.2 A99 - 2766 A28 - 2033.9 A50
+ 1903.8 A37 + 1816.9 A53 + 1731.1 A44 - 1925.9 A13 + 1873 A12
- 1417.7 A54 - 1419.4 A05 + 928 A25 + 744.3 A57 + 636.4 A60
- 647.2 A52 - 616.5 A58 + 707.9 A09 + 519.8 A45 - 400.1 A61
- 335.7 A81 + 310 A97 + 363.5 A07 + 355.2 A00 + 166.7 A90
Rule 4: [63 cases, mean 30.403, range 2.9 to 58.5, est err 1.727]
if
A05 <= 3.17348
A40 > 3.08971
then
Fat = 10.747 - 12872.2 A38 + 11360 A37 + 10884.5 A39 + 10841.4 A53
- 11492 A13 + 11176.6 A12 - 8549.5 A34 - 8459.4 A54 - 7322.5 A40
- 8778.5 A05 - 6452.5 A98 + 5477.3 A99 + 4441.1 A57 + 4477.9 A30
+ 4813.9 A09 + 3797.3 A60 - 3861.7 A52 - 3727.5 A95 - 4041.9 A28
- 3678.7 A58 + 3499.4 A44 + 3038.2 A36 - 2779.3 A61 - 2742.6 A48
+ 2451.8 A49 + 2121.4 A93 + 2116.7 A25 + 1849.6 A97 + 2169.1 A07
+ 2119.4 A00 - 1244.4 A45 - 1101.8 A50 + 994.6 A90 - 294.8 A16
- 229.5 A81
Despite the high dimensionality of this data, a ten-fold cross-validation shows a very good fit on the unseen cases:
The third example uses a dataset from the UCI Machine Learning Repository. The data, donated by Prof. I-Cheng Yeh, consist of information on 1,030 concrete samples showing, for each, the value of eight relevant properties and its compressive strength.
A model consisting of 21 rules is constructed in less than the same tenth of a second. Here are a few examples:
Rule 1: [60 cases, mean 19.478, range 6.27 to 41.64, est err 2.228]
if
Cement > 218.9
Water > 162.1
Age <= 3
then
Concrete compressive strength = 8.185 + 6.115 Age + 0.079 Cement
+ 0.82 Superplasticizer - 0.187 Water
+ 0.007 Blast Furnace Slag
- 0.003 Fine Aggregate + 0.003 Fly Ash
Rule 2: [94 cases, mean 20.290, range 2.33 to 49.25, est err 3.675]
if
Cement <= 218.9
Blast Furnace Slag > 76
Fly Ash <= 116
Superplasticizer <= 7.6
Age <= 28
then
Concrete compressive strength = -76.875 + 0.752 Age + 0.133 Cement
+ 0.94 Superplasticizer
+ 0.059 Blast Furnace Slag + 0.131 Water
+ 0.029 Fine Aggregate + 0.005 Fly Ash
+ 0.002 Coarse Aggregate
Rule 20: [106 cases, mean 59.542, range 31.72 to 82.6, est err 5.348]
if
Superplasticizer > 7.8
Age > 28
then
Concrete compressive strength = 76.053 + 0.14 Blast Furnace Slag
+ 0.1 Cement - 0.393 Water
+ 0.098 Fly Ash + 0.091 Age
- 0.75 Superplasticizer
Rule 21: [12 cases, mean 59.808, range 41.37 to 81.75, est err 13.248]
if
Cement > 218.9
Water <= 162.1
Superplasticizer <= 8.5
then
Concrete compressive strength = 49.519 - 2.03 Superplasticizer
+ 0.159 Age + 0.026 Cement
+ 0.02 Blast Furnace Slag
+ 0.015 Fly Ash - 0.032 Water
+ 0.003 Fine Aggregate
+ 0.002 Coarse Aggregate
A scatter-plot of the results on unseen cases from a ten-fold cross-validation shows that the model is not too bad.
The fourth example also uses a dataset from the Machine Learning Repository. The age of an abalone is determined by counting its rings, then adding 1.5. To quote the documentation with the dataset: "The age of abalone is determined by cutting the shell through the cone, staining it, and counting the number of rings through a microscope -- a boring and time-consuming task." The idea here is to use other more easily-obtained information to estimate the number of rings, and hence the age.
This dataset is divided into a training set of 2800 cases and a separate test set of 1376. Cubist finds this five-rule model from the training cases (you guessed it--in less than a tenth of a second):
Rule 1: [177 cases, mean 5.4, range 1 to 11, est err 0.9]
if
Shucked weight <= 0.057
then
Rings = 3.2 + 45.71 Whole weight - 61.5 Shucked weight + 5 Shell weight
+ 1.6 Diameter
Rule 2: [514 cases, mean 7.5, range 4 to 19, est err 1.1]
if
Sex = I
Shucked weight > 0.057
Shell weight <= 0.1685
then
Rings = 4.5 + 28.4 Shell weight - 11.9 Shucked weight
+ 2.62 Whole weight + 3.4 Diameter - 2 Viscera weight
+ 4.5 Height - 1.1 Length
Rule 3: [322 cases, mean 8.9, range 4 to 18, est err 1.4]
if
Sex in {M, F}
Shucked weight > 0.057
Shell weight <= 0.1685
then
Rings = 7 + 17.73 Whole weight - 36.9 Shucked weight + 14.1 Shell weight
- 10.6 Viscera weight + 0.5 Diameter + 0.9 Height
Rule 4: [916 cases, mean 11.4, range 6 to 27, est err 1.9]
if
Shucked weight <= 0.4445
Shell weight > 0.1685
then
Rings = 11.5 + 20.09 Whole weight - 30.6 Shucked weight
- 17.1 Viscera weight - 12 Length + 8.6 Shell weight
+ 2.8 Height + 0.4 Diameter
Rule 5: [871 cases, mean 11.4, range 6 to 29, est err 1.6]
if
Shucked weight > 0.4445
Shell weight > 0.1685
then
Rings = 7.6 + 7.45 Whole weight - 16.3 Shucked weight
+ 10.7 Shell weight - 3.8 Viscera weight - 2.6 Length
+ 6.9 Height + 1.4 Diameter
The plot of predicted versus actual values on the remaining 1376 unseen cases shows a reasonable level of agreement:
This example also comes from Statlib. The data, contributed by Ian McLeod, concern 946 successive mean monthly flows of the Fraser River at Hope, B.C.
The goal in this application is to predict the flow in a particular month in terms of the flows for previous months. In this example, we will use the previous 20 months' mean flows: there are thus 926 cases described by 20 independent attributes and the target attribute, all continuous values.
Cubist finds five rules from the 926 cases (in 0.1 seconds):
Rule 1: [591 cases, mean 1399.5, range 482 to 4460, est err 249.6]
if
[-12 months] <= 2640
then
This month = 128.3 + 0.65 [-1 month] + 0.205 [-11 months]
- 0.15 [-2 months] + 0.072 [-3 months] - 0.056 [-13 months]
+ 0.047 [-12 months] - 0.011 [-10 months]
- 0.01 [-14 months]
Rule 2: [83 cases, mean 3212.5, range 1300 to 5460, est err 332.6]
if
[-15 months] > 4010
[-12 months] > 2640
then
This month = 1435.2 + 0.47 [-1 month] - 0.076 [-15 months]
+ 0.037 [-12 months] - 0.03 [-9 months] - 0.03 [-2 months]
+ 0.029 [-4 months] + 0.025 [-8 months]
+ 0.024 [-11 months] - 0.015 [-5 months]
- 0.015 [-13 months] - 0.011 [-14 months]
Rule 3: [13 cases, mean 4185.1, range 896 to 6550, est err 856.0]
if
[-12 months] > 2640
[-1 month] <= 1050
then
This month = -4938.5 + 8.5 [-1 month] + 0.349 [-10 months]
- 0.093 [-9 months] + 0.076 [-8 months] - 0.07 [-2 months]
- 0.053 [-15 months] + 0.041 [-11 months]
Rule 4: [68 cases, mean 4932.4, range 1220 to 8170, est err 845.9]
if
[-15 months] <= 4010
[-13 months] <= 2780
[-12 months] > 2640
[-1 month] > 1050
then
This month = 5006.4 - 1.254 [-17 months] - 0.792 [-13 months]
+ 0.68 [-18 months] + 0.47 [-1 month] + 0.293 [-12 months]
- 0.228 [-8 months] - 0.044 [-9 months] - 0.03 [-2 months]
+ 0.018 [-11 months] - 0.018 [-15 months]
Rule 5: [171 cases, mean 5963.9, range 2080 to 10800, est err 812.7]
if
[-15 months] <= 4010
[-13 months] > 2780
[-12 months] > 2640
then
This month = 3876.3 + 0.796 [-8 months] - 0.792 [-9 months]
+ 0.51 [-1 month] + 0.426 [-10 months] - 0.42 [-2 months]
- 0.333 [-15 months] - 0.032 [-5 months]
+ 0.026 [-12 months] + 0.019 [-11 months]
- 0.015 [-14 months] + 0.015 [-3 months]
- 0.011 [-13 months]
A scatter-plot of the results of a ten-fold cross-validation shows a reasonably high level of agreement between actual and predicted flows for the unseen cases.
The default "persistence" model obtained by always predicting the previous month's flow explains only 45% of the variance for unseen cases, noticeably less than the 88% explained by the Cubist model.
This example originates at the Pacific Marine Environmental Laboratory in the National Oceanic and Atmospheric Administration of the US Department of Commerce, and can be found among the datasets in the UCI KDD Archive. There are 178,080 observations taken between 1980 and 1998 that record values for the following:
Abbrev Attribute Case 1 Case 2 Case 3 ..... year year of observation 80 80 80 month month ditto 3 3 3 day day of month 7 8 9 latitude latitude of buoy -0.02 -0.02 -0.02 longitude longitude ditto -109.46 -109.46 -109.46 zon.winds zonal winds (M/S) -6.8 -4.9 -4.5 mer.winds meridional winds (M/S) 0.7 1.1 2.2 humidity relative humidity (%) ? ? ? air temp air temperature 23.5 23.6 23.62 s.s.temp sea surface temperature 24.01 23.91 23.82
Here we will attempt to predict the value of the sea surface temperature from the values of the other attributes.
The cases were divided into training and test sets containing 89,040 cases each, and Cubist took 1.6 seconds to build a model containing 100 rules (the default maximum number). Two rules each for the lowest and highest sea temperatures should illustrate the idea:
Rule 1: [1438 cases, mean 23.430, range 18.87 to 29.69, est err 0.281]
if
year <= 96
latitude <= -1.99
longitude > -109.66
longitude <= -94.7
then
s.s.temp = -83.4 - 0.8999 longitude + 1.02 air temp - 0.144 latitude
- 0.037 year - 0.029 mer.winds + 0.008 humidity
+ 0.01 zon.winds
Rule 2: [317 cases, mean 23.467, range 19.37 to 27.22, est err 0.874]
if
year <= 94
month > 4
latitude > -0.1
latitude <= 5.31
longitude > -139.9
longitude <= -95.15
air temp > 26.86
air temp <= 26.869
then
s.s.temp = 17.749 + 1.217 latitude - 0.415 month + 0.296 mer.winds
- 0.1 zon.winds + 0.097 year - 0.016 day - 0.005 humidity
Rule 99: [506 cases, mean 29.676, range 26.81 to 30.98, est err 0.296]
if
latitude <= -0.57
longitude > -95.15
air temp > 26.86
air temp <= 26.869
then
s.s.temp = 28.631 + 0.0087 longitude + 0.097 latitude - 0.03 mer.winds
- 0.016 zon.winds
Rule 100: [885 cases, mean 29.688, range 27.4 to 31.17, est err 0.293]
if
year > 93
year <= 96
latitude <= 6.68
longitude > -124.98
zon.winds > -0.5
mer.winds <= -0.3
air temp > 26.869
then
s.s.temp = 1.7 + 0.0056 longitude + 0.202 year + 0.29 air temp
+ 0.058 mer.winds - 0.046 zon.winds - 0.026 latitude
The scatter-plot for the unseen test cases is fairly dense since it contains nearly 90,000 points, but the correlation coefficient (.98) shows that the model is not bad:
Since these examples were all run using Cubist's default parameter settings, they do not illustrate several additional capabilities:
Please see the tutorial for more details.
| © RULEQUEST RESEARCH 2010 | Last updated February 2010 |
| home | products | download | evaluations | prices | purchase | contact us |