rF() func
, , sample = #1
sex
-income Female Male
- <=50K 28.17199 43.5344
- >50K 6.83616 21.4575
+income Female Male
+ <=50K 27.6411 43.5611
+ >50K 6.9703 21.8275
, , sample = #2
sex
income Female Male
- <=50K 28.18455 43.4979
- >50K 7.24585 21.0717
+ <=50K 28.31471 43.1568
+ >50K 7.21764 21.3108
, , sample = #3
sex
income Female Male
- <=50K 28.13035 43.6037
- >50K 7.00727 21.2587These possible full-population frequency distributions can be used to assess how much the probabilities we find could change, if we collected a much, much larger amount of training data. Here is an example:
@@ -1037,20 +1037,20 @@rF() func
sex
occupation Female Male
- Adm-clerical 0.29 0.32
- Armed-Forces 0.16 0.17
- Craft-repair 0.17 0.47
- Exec-managerial 0.31 0.62
- Farming-fishing 0.16 0.23
- Handlers-cleaners 0.17 0.19
- Machine-op-inspct 0.17 0.29
- Other-service 0.20 0.22
- Priv-house-serv 0.16 0.16
- Prof-specialty 0.36 0.56
- Protective-serv 0.16 0.28
- Sales 0.22 0.49
- Tech-support 0.21 0.30
- Transport-moving 0.16 0.31the agent believes (at around 68%) that the current probability wouldn’t change more than about ±0.5%.
@@ -1099,15 +1099,14 @@marital_status relationship education occupation workclass
0.10074130 0.09046621 0.06332052 0.05506897 0.03002995
@@ -1177,7 +1176,7 @@ ## Print total time required
print(Sys.time() - stopwatch)
Time difference of 4.23584 secsTime difference of 4.21148 secs## Histogram and average accuracy (rounded to one decimal)
myhist(testprobs, n = seq(0,1,length.out = 10), plot = TRUE,
diff --git a/docs/example_opm1_files/figure-html/unnamed-chunk-10-1.png b/docs/example_opm1_files/figure-html/unnamed-chunk-10-1.png
index 4c7f29a..6b4394e 100644
Binary files a/docs/example_opm1_files/figure-html/unnamed-chunk-10-1.png and b/docs/example_opm1_files/figure-html/unnamed-chunk-10-1.png differ
diff --git a/docs/example_opm1_files/figure-html/unnamed-chunk-11-1.png b/docs/example_opm1_files/figure-html/unnamed-chunk-11-1.png
index ea3b722..3a3eae2 100644
Binary files a/docs/example_opm1_files/figure-html/unnamed-chunk-11-1.png and b/docs/example_opm1_files/figure-html/unnamed-chunk-11-1.png differ
diff --git a/docs/example_opm1_files/figure-html/unnamed-chunk-12-1.png b/docs/example_opm1_files/figure-html/unnamed-chunk-12-1.png
index 412695f..9327988 100644
Binary files a/docs/example_opm1_files/figure-html/unnamed-chunk-12-1.png and b/docs/example_opm1_files/figure-html/unnamed-chunk-12-1.png differ
diff --git a/docs/example_opm1_files/figure-html/unnamed-chunk-13-1.png b/docs/example_opm1_files/figure-html/unnamed-chunk-13-1.png
index 7bd149c..cbfc950 100644
Binary files a/docs/example_opm1_files/figure-html/unnamed-chunk-13-1.png and b/docs/example_opm1_files/figure-html/unnamed-chunk-13-1.png differ
diff --git a/docs/example_opm1_files/figure-html/unnamed-chunk-14-1.png b/docs/example_opm1_files/figure-html/unnamed-chunk-14-1.png
index bb01ee1..b811b7d 100644
Binary files a/docs/example_opm1_files/figure-html/unnamed-chunk-14-1.png and b/docs/example_opm1_files/figure-html/unnamed-chunk-14-1.png differ
diff --git a/docs/example_opm1_files/figure-html/unnamed-chunk-20-1.png b/docs/example_opm1_files/figure-html/unnamed-chunk-20-1.png
index 5191d75..a37ff5a 100644
Binary files a/docs/example_opm1_files/figure-html/unnamed-chunk-20-1.png and b/docs/example_opm1_files/figure-html/unnamed-chunk-20-1.png differ
diff --git a/docs/example_opm1_files/figure-html/unnamed-chunk-9-1.png b/docs/example_opm1_files/figure-html/unnamed-chunk-9-1.png
index c61bb6a..196334a 100644
Binary files a/docs/example_opm1_files/figure-html/unnamed-chunk-9-1.png and b/docs/example_opm1_files/figure-html/unnamed-chunk-9-1.png differ
diff --git a/docs/search.json b/docs/search.json
index b9be1f6..5362b73 100644
--- a/docs/search.json
+++ b/docs/search.json
@@ -1978,7 +1978,7 @@
"href": "example_opm1.html#application-and-exploration",
"title": "35 Example application: adult-income task",
"section": "35.5 Application and exploration",
- "text": "35.5 Application and exploration\n\nApplication: only predictands\nOur two agents are ready to be applied to new instances.\nBefore applying them, let’s check some of their inferences, and see if we find anything unconvincing about them. If we find something unconvincing, it means that the background information we provided to the agent doesn’t match the one in our intuition. Then there are two or three possibilities: our intuition is misleading us and need correcting; or we need to go back to stage Collect & prepare background info and correct the background information given to the agent; or a combination of these two possibilities.\nWe ask the opm10 agent to forecast the \\(\\mathit{income}\\) of the next unit, using the infer() function:\n\ninfer(agent = opm10, predictand = 'income')\n\nincome\n <=50K >50K \n0.506288 0.493712 \n\n\nThis agent gives a slightly larger probability to the \\({\\small\\verb;<=50K;}\\) case. Using the function plotFsamples1D() we can also inspect the opm10-agent’s belief about the frequency distribution of \\(\\mathit{income}\\) for the full population. This belief is represented by a generalized scatter plot of 200 representative frequency distributions, represented as the light-blue lines:\n\nplotFsamples1D(agent = opm10,\n n = 200, # number of example frequency distributions\n predictand = 'income',\n ylim = c(0,1), # y-axis range\n main = 'opm10') # plot title\n\n\n\n\n\n\n\n\nwhere the black line is the probability distribution previously calculated with the infer() function.\nThis plot expresses the opm10-agent’s belief that future training data might lead to even higher probabilities for \\({\\small\\verb;<=50K;}\\). But note that the agent is not excluding the possibility of lower probabilities.\nLet’s visualize the beliefs of the opmall-agent, trained with the full training dataset:\n\nplotFsamples1D(agent = opmall, n = 200, predictand = 'income',\n ylim = c(0,1), main = 'opmall')\n\n\n\n\n\n\n\n\nThe probability that the next unit has \\(\\mathit{income}\\mathclose{}\\mathord{\\nonscript\\mkern 0mu\\textrm{\\small=}\\nonscript\\mkern 0mu}\\mathopen{}{\\small\\verb;<=50;}\\) is now above 70%. Also note that the opmall-agent doesn’t believe that this probability would change appreciably if more training data were provided.\n\n\nWe can perform a similar exploration for any other variate. Let’s take the \\(\\mathit{race}\\) variate for example:\n\nplotFsamples1D(agent = opm10, n = 200, predictand = 'race',\n ylim = c(0,1), main = 'opm10',\n cex.axis = 0.75) # smaller axis-font size\n\n\n\n\n\n\n\n\nNote again how the little-trained opm10-agent has practically uniform beliefs. But it’s also expressing the fact that future training data will probably increase the probability of \\(\\mathit{race}\\mathclose{}\\mathord{\\nonscript\\mkern 0mu\\textrm{\\small=}\\nonscript\\mkern 0mu}\\mathopen{}{\\small\\verb;White;}\\).\nThis is corroborated by the fully-trained agent:\n\nplotFsamples1D(agent = opmall, n = 200, predictand = 'race',\n ylim = c(0,1), main = 'opmall', cex.axis = 0.75)\n\n\n\n\n\n\n\n\n\n\nThese checks are satisfying, but it’s good to examine their agreement or disagreement with our intuition. Examine the last plot for example. The opmall agent has very firm beliefs (no spread in the light-blue lines) about the full-population distribution of \\(\\mathit{race}\\). Do you think its beliefs are too firm, after 11 000 datapoints? would you like the agent to be more “open-minded”? In that case you should go back to the Collect & prepare background info stage, and for example modify the parameters \\(k_{\\text{mi}}, k_{\\text{ma}}\\), then re-check. Or you could even try an agent with a different initial belief distribution.\nIn making this kind of considerations it’s important to keep in mind what we learned and observed in previous chapters:\n\n\n\n\n\n\nNote\n\n\n\nOur goal: optimality, not “success”\nRemember (§ 2.3 and § 8.1) that a probability represents the rational degree of belief that an agent should have given the particular information available. We can’t judge a probability from the value it assigns to something we later learn to be true – because according to the information available it could be more rational (and optimal) to consider that something implausible (recall the example in § 8.1 of an object falling from the sky as we cross the street).\nFrom this point of view we should be wary of comparing the probability of something with our a-posteriori knowledge about it.\n\n\nThe data cannot speak for themselves\nWe could build an agent that remains more “open-minded” (more spread in the light-blue lines), having received exactly the same training data. This “open-mindedness” therefore cannot be determined by the training data. Once more this shows that data cannot “speak for themselves” (§ 27.4).\n\n\n\n\n\n\n\n\n\nApplication: specifying predictors\nLet’s now draw inferences by specifying some predictors.\nWe ask the opm10 agent to forecast the \\(\\mathit{income}\\) of a new unit, given that the unit is known to have \\(\\mathit{occupation}\\mathclose{}\\mathord{\\nonscript\\mkern 0mu\\textrm{\\small=}\\nonscript\\mkern 0mu}\\mathopen{}{\\small\\verb;Exec-managerial;}\\) and \\(\\mathit{sex}\\mathclose{}\\mathord{\\nonscript\\mkern 0mu\\textrm{\\small=}\\nonscript\\mkern 0mu}\\mathopen{}{\\small\\verb;Male;}\\) (two predictor variates). What would you expect?\nThe opm10-agent’s belief about the unit – as well as about the full subpopulation (§ 22.1) of units having those predictors – is shown in the following plot:\n\nplotFsamples1D(agent = opm10, n = 200,\n predictand = 'income',\n predictor = list(occupation = 'Exec-managerial',\n sex = 'Male'),\n ylim = c(0,1), main = 'opm10')\n\n\n\n\n\n\n\n\nNote how the opm10-agent still slightly higher probability to \\(\\mathit{income}\\mathclose{}\\mathord{\\nonscript\\mkern 0mu\\textrm{\\small=}\\nonscript\\mkern 0mu}\\mathopen{}{\\small\\verb;<=50;}\\), but at the same time it is quite uncertain about the subpopulation frequencies; more than if the predictor had not been specified. That is, according to this little-trained agent there could be large variety of possibilities within this specific subpopulation.\nThe opmall-agent’s beliefs are shown below:\n\nplotFsamples1D(agent = opmall, n = 200,\n predictand = 'income',\n predictor = list(occupation = 'Exec-managerial',\n sex = 'Male'),\n ylim = c(0,1), main = 'opmall')\n\n\n\n\n\n\n\n\nit believes with around 55% probability that such a unit would have higher, \\({\\small\\verb;>50K;}\\) income. The representative subpopulation-frequency distributions in light-blue indicate that this belief is unlikely to be changed by new training data.\n\n\nLet’s now see an example of our agent’s versatility by switching predictands and predictors. We tell the opmall-agent that the new unit has \\(\\mathit{income}\\mathclose{}\\mathord{\\nonscript\\mkern 0mu\\textrm{\\small=}\\nonscript\\mkern 0mu}\\mathopen{}{\\small\\verb;>50;}\\), and ask it to infer the joint variate \\((\\mathit{occupation} \\mathbin{\\mkern-0.5mu,\\mkern-0.5mu}\\mathit{sex})\\); let’s present the results in rounded percentages:\n\nresult <- infer(agent = opmall,\n predictand = c('occupation', 'sex'),\n predictor = list(income = '>50K'))\n\nround(result * 100, 1) # round to one decimal\n\n sex\noccupation Female Male\n Adm-clerical 3.1 4.1\n Armed-Forces 1.0 1.0\n Craft-repair 1.1 9.4\n Exec-managerial 3.6 16.0\n Farming-fishing 1.0 2.1\n Handlers-cleaners 1.0 1.6\n Machine-op-inspct 1.1 3.1\n Other-service 1.6 1.8\n Priv-house-serv 1.0 1.0\n Prof-specialty 4.8 14.7\n Protective-serv 1.0 3.2\n Sales 1.8 10.1\n Tech-support 1.5 3.4\n Transport-moving 1.1 3.7\n\n\nIt returns a 14 × 2 table of joint probabilities. The most probable combinations are \\(({\\small\\verb;Exec-managerial;}, {\\small\\verb;Male;})\\) and \\(({\\small\\verb;Prof-specialty;}, {\\small\\verb;Male;})\\).\n\n\nThe rF() function\nThis function generates full-population frequency distributions (even for subpopulations) that are probable according to the data. It is used internally by plotFsamples1D(), which plots the generated frequency distributions as light-blue lines.\nLet’s see, as an example, three samples of how the full-population frequency distribution for \\(\\mathit{sex} \\mathbin{\\mkern-0.5mu,\\mkern-0.5mu}\\mathit{income}\\) (jointly) could be:\n\nresult <- rF(n = 3, # number of samples\n agent = opmall, \n predictand = c('sex', 'income'))\n\n## name the samples\ndimnames(result)[1] <- list(samples = paste0('#',1:3))\n\n## permute & print so that samples are the last array dimension\nprint(aperm(result) * 100)\n\n, , sample = #1\n\n sex\nincome Female Male\n <=50K 28.17199 43.5344\n >50K 6.83616 21.4575\n\n, , sample = #2\n\n sex\nincome Female Male\n <=50K 28.18455 43.4979\n >50K 7.24585 21.0717\n\n, , sample = #3\n\n sex\nincome Female Male\n <=50K 28.13035 43.6037\n >50K 7.00727 21.2587\n\n\nThese possible full-population frequency distributions can be used to assess how much the probabilities we find could change, if we collected a much, much larger amount of training data. Here is an example:\nWe generate 1000 frequency distributions for \\((\\mathit{occupation} \\mathbin{\\mkern-0.5mu,\\mkern-0.5mu}\\mathit{sex})\\) given \\(\\mathit{income}\\mathclose{}\\mathord{\\nonscript\\mkern 0mu\\textrm{\\small=}\\nonscript\\mkern 0mu}\\mathopen{}{\\small\\verb;>50K;}\\), and then take the standard deviations of the samples as a rough measure of how much the probabilities we calculated a couple of cells above could change:\n\nfreqsamples <- rF(n = 1000,\n agent = opmall,\n predictand = c('occupation', 'sex'),\n predictor = list(income = '>50K'))\n\nvariability <- apply(freqsamples,\n c('occupation','sex'), # which dimensions to apply\n sd) # function to apply to those dimensions\n\nround(variability * 100, 2) # round to two decimals\n\n sex\noccupation Female Male\n Adm-clerical 0.29 0.32\n Armed-Forces 0.16 0.17\n Craft-repair 0.17 0.47\n Exec-managerial 0.31 0.62\n Farming-fishing 0.16 0.23\n Handlers-cleaners 0.17 0.19\n Machine-op-inspct 0.17 0.29\n Other-service 0.20 0.22\n Priv-house-serv 0.16 0.16\n Prof-specialty 0.36 0.56\n Protective-serv 0.16 0.28\n Sales 0.22 0.49\n Tech-support 0.21 0.30\n Transport-moving 0.16 0.31\n\n\nthe agent believes (at around 68%) that the current probability wouldn’t change more than about ±0.5%.\n\n\nThe inferences above were partially meant as checks, but we see that we can actually ask our agent a wide variety of questions about the full population, and do all sorts of association studies.\n\n\n\n\n\n\n No “test” or “validation” datasets used or needed\n\n\n\nThe tests and explorations above were done without any “validation” or “test” datasets. This is because our agent is capable of calculating and showing its beliefs about the full population – and therefore about future data.\nThe need for validation or test datasets with common machine-learning algorithms arise from the fact that full-population beliefs are hidden or, more commonly, not computed at all, in order to gain speed. The application of the trained machine-learning algorithm to a validation dataset is an approximate way of extracting such beliefs.\n\n\n\n\n\n\nExploring the population properties: mutual information\nIn § 18.5 we introduced mutual information as the information-theoretic measure of mutual relevance and association of two quantities or variates. For the present task, the opmall-agent can tell us the mutual information between any two sets of variates of our choice, with the function mutualinfo().\nFor instance, let’s calculate the mutual information between \\(\\mathit{occupation}\\) and \\(\\mathit{marital\\_status}\\).\n\nmutualinfo(agent = opmall,\n A = 'occupation', B = 'marital_status')\n\n[1] 0.0827823\n\n\nIt is a very low association: knowing either variate decreases the effective number of possible values of the other only \\(2^{0.0827823\\,\\mathit{Sh}} \\approx 1.06\\) times.\nNow let’s consider a scenario where, in order to save resources, we can use only one variate to infer the income. Which of the other variates should we prefer? We can calculate the mutual information between each of them, in turn, and \\(\\mathit{income}\\):\n\n## list of all variates\nvariates <- names(dimnames(opmall$counts))\n\n## list of all variates except 'income'\npredictors <- variates[variates != 'income']\n\n## prepare vector to contain the mutual information\nrelevances <- numeric(length(predictors))\nnames(relevances) <- predictors\n\n## calculate, for each variate,\n## the joint probability distribution 'probs'\n## and then mutual information 'relevance' (in shannons)\n## between 'income' and that variate\nfor(var in predictors){\n relevances[var] <- mutualinfo(agent = opmall, A = 'income', B = var)\n}\n\n## output the mutual informations in decreasing order\nsort(relevances, decreasing = TRUE)\n\nmarital_status relationship education occupation workclass \n 0.10074130 0.09046621 0.06332052 0.05506897 0.03002995 \nnative_country sex race \n 0.01925227 0.01456655 0.00870089 \n\n\nIf we had to choose only one variate to infer the outcome, on average it would be best to use \\(\\mathit{marital\\_status}\\). Our last choice should be \\(\\mathit{race}\\).\n\n\n\n\n\n\n Exercise\n\n\n\nNow consider the scenario where we must exclude one variate from the eight predictors, or, equivalently, we can only use seven variates as predictors. Which variate should we exclude?\nPrepare a script similar to the one above: it calculates the mutual information between \\(\\mathit{income}\\) and the other predictors but with one omitted, omitting each of the eight in turn.\nWarning: this computation might require 10 or more minutes to complete.\n\nWhich single variate should not be omitted from the predictors? which single variate could be dropped?\nDo you obtain the same relevance ranking as in the “use-one-variate-only” scenario above?",
+ "text": "35.5 Application and exploration\n\nApplication: only predictands\nOur two agents are ready to be applied to new instances.\nBefore applying them, let’s check some of their inferences, and see if we find anything unconvincing about them. If we find something unconvincing, it means that the background information we provided to the agent doesn’t match the one in our intuition. Then there are two or three possibilities: our intuition is misleading us and need correcting; or we need to go back to stage Collect & prepare background info and correct the background information given to the agent; or a combination of these two possibilities.\nWe ask the opm10 agent to forecast the \\(\\mathit{income}\\) of the next unit, using the infer() function:\n\ninfer(agent = opm10, predictand = 'income')\n\nincome\n <=50K >50K \n0.506288 0.493712 \n\n\nThis agent gives a slightly larger probability to the \\({\\small\\verb;<=50K;}\\) case. Using the function plotFsamples1D() we can also inspect the opm10-agent’s belief about the frequency distribution of \\(\\mathit{income}\\) for the full population. This belief is represented by a generalized scatter plot of 200 representative frequency distributions, represented as the light-blue lines:\n\nplotFsamples1D(agent = opm10,\n n = 200, # number of example frequency distributions\n predictand = 'income',\n ylim = c(0,1), # y-axis range\n main = 'opm10') # plot title\n\n\n\n\n\n\n\n\nwhere the black line is the probability distribution previously calculated with the infer() function.\nThis plot expresses the opm10-agent’s belief that future training data might lead to even higher probabilities for \\({\\small\\verb;<=50K;}\\). But note that the agent is not excluding the possibility of lower probabilities.\nLet’s visualize the beliefs of the opmall-agent, trained with the full training dataset:\n\nplotFsamples1D(agent = opmall, n = 200, predictand = 'income',\n ylim = c(0,1), main = 'opmall')\n\n\n\n\n\n\n\n\nThe probability that the next unit has \\(\\mathit{income}\\mathclose{}\\mathord{\\nonscript\\mkern 0mu\\textrm{\\small=}\\nonscript\\mkern 0mu}\\mathopen{}{\\small\\verb;<=50;}\\) is now above 70%. Also note that the opmall-agent doesn’t believe that this probability would change appreciably if more training data were provided.\n\n\nWe can perform a similar exploration for any other variate. Let’s take the \\(\\mathit{race}\\) variate for example:\n\nplotFsamples1D(agent = opm10, n = 200, predictand = 'race',\n ylim = c(0,1), main = 'opm10',\n cex.axis = 0.75) # smaller axis-font size\n\n\n\n\n\n\n\n\nNote again how the little-trained opm10-agent has practically uniform beliefs. But it’s also expressing the fact that future training data will probably increase the probability of \\(\\mathit{race}\\mathclose{}\\mathord{\\nonscript\\mkern 0mu\\textrm{\\small=}\\nonscript\\mkern 0mu}\\mathopen{}{\\small\\verb;White;}\\).\nThis is corroborated by the fully-trained agent:\n\nplotFsamples1D(agent = opmall, n = 200, predictand = 'race',\n ylim = c(0,1), main = 'opmall', cex.axis = 0.75)\n\n\n\n\n\n\n\n\n\n\nThese checks are satisfying, but it’s good to examine their agreement or disagreement with our intuition. Examine the last plot for example. The opmall agent has very firm beliefs (no spread in the light-blue lines) about the full-population distribution of \\(\\mathit{race}\\). Do you think its beliefs are too firm, after 11 000 datapoints? would you like the agent to be more “open-minded”? In that case you should go back to the Collect & prepare background info stage, and for example modify the parameters \\(k_{\\text{mi}}, k_{\\text{ma}}\\), then re-check. Or you could even try an agent with a different initial belief distribution.\nIn making this kind of considerations it’s important to keep in mind what we learned and observed in previous chapters:\n\n\n\n\n\n\nNote\n\n\n\nOur goal: optimality, not “success”\nRemember (§ 2.3 and § 8.1) that a probability represents the rational degree of belief that an agent should have given the particular information available. We can’t judge a probability from the value it assigns to something we later learn to be true – because according to the information available it could be more rational (and optimal) to consider that something implausible (recall the example in § 8.1 of an object falling from the sky as we cross the street).\nFrom this point of view we should be wary of comparing the probability of something with our a-posteriori knowledge about it.\n\n\nThe data cannot speak for themselves\nWe could build an agent that remains more “open-minded” (more spread in the light-blue lines), having received exactly the same training data. This “open-mindedness” therefore cannot be determined by the training data. Once more this shows that data cannot “speak for themselves” (§ 27.4).\n\n\n\n\n\n\n\n\n\nApplication: specifying predictors\nLet’s now draw inferences by specifying some predictors.\nWe ask the opm10 agent to forecast the \\(\\mathit{income}\\) of a new unit, given that the unit is known to have \\(\\mathit{occupation}\\mathclose{}\\mathord{\\nonscript\\mkern 0mu\\textrm{\\small=}\\nonscript\\mkern 0mu}\\mathopen{}{\\small\\verb;Exec-managerial;}\\) and \\(\\mathit{sex}\\mathclose{}\\mathord{\\nonscript\\mkern 0mu\\textrm{\\small=}\\nonscript\\mkern 0mu}\\mathopen{}{\\small\\verb;Male;}\\) (two predictor variates). What would you expect?\nThe opm10-agent’s belief about the unit – as well as about the full subpopulation (§ 22.1) of units having those predictors – is shown in the following plot:\n\nplotFsamples1D(agent = opm10, n = 200,\n predictand = 'income',\n predictor = list(occupation = 'Exec-managerial',\n sex = 'Male'),\n ylim = c(0,1), main = 'opm10')\n\n\n\n\n\n\n\n\nNote how the opm10-agent still slightly higher probability to \\(\\mathit{income}\\mathclose{}\\mathord{\\nonscript\\mkern 0mu\\textrm{\\small=}\\nonscript\\mkern 0mu}\\mathopen{}{\\small\\verb;<=50;}\\), but at the same time it is quite uncertain about the subpopulation frequencies; more than if the predictor had not been specified. That is, according to this little-trained agent there could be large variety of possibilities within this specific subpopulation.\nThe opmall-agent’s beliefs are shown below:\n\nplotFsamples1D(agent = opmall, n = 200,\n predictand = 'income',\n predictor = list(occupation = 'Exec-managerial',\n sex = 'Male'),\n ylim = c(0,1), main = 'opmall')\n\n\n\n\n\n\n\n\nit believes with around 55% probability that such a unit would have higher, \\({\\small\\verb;>50K;}\\) income. The representative subpopulation-frequency distributions in light-blue indicate that this belief is unlikely to be changed by new training data.\n\n\nLet’s now see an example of our agent’s versatility by switching predictands and predictors. We tell the opmall-agent that the new unit has \\(\\mathit{income}\\mathclose{}\\mathord{\\nonscript\\mkern 0mu\\textrm{\\small=}\\nonscript\\mkern 0mu}\\mathopen{}{\\small\\verb;>50;}\\), and ask it to infer the joint variate \\((\\mathit{occupation} \\mathbin{\\mkern-0.5mu,\\mkern-0.5mu}\\mathit{sex})\\); let’s present the results in rounded percentages:\n\nresult <- infer(agent = opmall,\n predictand = c('occupation', 'sex'),\n predictor = list(income = '>50K'))\n\nround(result * 100, 1) # round to one decimal\n\n sex\noccupation Female Male\n Adm-clerical 3.1 4.1\n Armed-Forces 1.0 1.0\n Craft-repair 1.1 9.4\n Exec-managerial 3.6 16.0\n Farming-fishing 1.0 2.1\n Handlers-cleaners 1.0 1.6\n Machine-op-inspct 1.1 3.1\n Other-service 1.6 1.8\n Priv-house-serv 1.0 1.0\n Prof-specialty 4.8 14.7\n Protective-serv 1.0 3.2\n Sales 1.8 10.1\n Tech-support 1.5 3.4\n Transport-moving 1.1 3.7\n\n\nIt returns a 14 × 2 table of joint probabilities. The most probable combinations are \\(({\\small\\verb;Exec-managerial;}, {\\small\\verb;Male;})\\) and \\(({\\small\\verb;Prof-specialty;}, {\\small\\verb;Male;})\\).\n\n\nThe rF() function\nThis function generates full-population frequency distributions (even for subpopulations) that are probable according to the data. It is used internally by plotFsamples1D(), which plots the generated frequency distributions as light-blue lines.\nLet’s see, as an example, three samples of how the full-population frequency distribution for \\(\\mathit{sex} \\mathbin{\\mkern-0.5mu,\\mkern-0.5mu}\\mathit{income}\\) (jointly) could be:\n\nresult <- rF(n = 3, # number of samples\n agent = opmall, \n predictand = c('sex', 'income'))\n\n## name the samples\ndimnames(result)[1] <- list(samples = paste0('#',1:3))\n\n## permute & print so that samples are the last array dimension\nprint(aperm(result) * 100)\n\n, , sample = #1\n\n sex\nincome Female Male\n <=50K 27.6411 43.5611\n >50K 6.9703 21.8275\n\n, , sample = #2\n\n sex\nincome Female Male\n <=50K 28.31471 43.1568\n >50K 7.21764 21.3108\n\n, , sample = #3\n\n sex\nincome Female Male\n <=50K 28.44690 42.0664\n >50K 7.28531 22.2014\n\n\nThese possible full-population frequency distributions can be used to assess how much the probabilities we find could change, if we collected a much, much larger amount of training data. Here is an example:\nWe generate 1000 frequency distributions for \\((\\mathit{occupation} \\mathbin{\\mkern-0.5mu,\\mkern-0.5mu}\\mathit{sex})\\) given \\(\\mathit{income}\\mathclose{}\\mathord{\\nonscript\\mkern 0mu\\textrm{\\small=}\\nonscript\\mkern 0mu}\\mathopen{}{\\small\\verb;>50K;}\\), and then take the standard deviations of the samples as a rough measure of how much the probabilities we calculated a couple of cells above could change:\n\nfreqsamples <- rF(n = 1000,\n agent = opmall,\n predictand = c('occupation', 'sex'),\n predictor = list(income = '>50K'))\n\nvariability <- apply(freqsamples,\n c('occupation','sex'), # which dimensions to apply\n sd) # function to apply to those dimensions\n\nround(variability * 100, 2) # round to two decimals\n\n sex\noccupation Female Male\n Adm-clerical 0.28 0.33\n Armed-Forces 0.15 0.16\n Craft-repair 0.17 0.48\n Exec-managerial 0.29 0.59\n Farming-fishing 0.17 0.23\n Handlers-cleaners 0.16 0.20\n Machine-op-inspct 0.17 0.28\n Other-service 0.21 0.23\n Priv-house-serv 0.17 0.16\n Prof-specialty 0.35 0.59\n Protective-serv 0.17 0.29\n Sales 0.21 0.50\n Tech-support 0.19 0.29\n Transport-moving 0.17 0.30\n\n\nthe agent believes (at around 68%) that the current probability wouldn’t change more than about ±0.5%.\n\n\nThe inferences above were partially meant as checks, but we see that we can actually ask our agent a wide variety of questions about the full population, and do all sorts of association studies.\n\n\n\n\n\n\n No “test” or “validation” datasets used or needed\n\n\n\nThe tests and explorations above were done without any “validation” or “test” datasets. This is because our agent is capable of calculating and showing its beliefs about the full population – and therefore about future data.\nThe need for validation or test datasets with common machine-learning algorithms arise from the fact that full-population beliefs are hidden or, more commonly, not computed at all, in order to gain speed. The application of the trained machine-learning algorithm to a validation dataset is an approximate way of extracting such beliefs.\n\n\n\n\n\n\nExploring the population properties: mutual information\nIn § 18.5 we introduced mutual information as the information-theoretic measure of mutual relevance and association of two quantities or variates. For the present task, the opmall-agent can tell us the mutual information between any two sets of variates of our choice, with the function mutualinfo().\nFor instance, let’s calculate the mutual information between \\(\\mathit{occupation}\\) and \\(\\mathit{marital\\_status}\\).\n\nmutualinfo(agent = opmall,\n A = 'occupation', B = 'marital_status')\n\n[1] 0.0827823\n\n\nIt is a very low association: knowing either variate decreases the effective number of possible values of the other only \\(2^{0.0827823\\,\\mathit{Sh}} \\approx 1.06\\) times.\nNow let’s consider a scenario where, in order to save resources, we can use only one variate to infer the income. Which of the other variates should we prefer? We can calculate the mutual information between each of them, in turn, and \\(\\mathit{income}\\):\n\n## list of all variates\nvariates <- names(dimnames(opmall$counts))\n\n## list of all variates except 'income'\npredictors <- variates[variates != 'income']\n\n## prepare vector to contain the mutual information\nrelevances <- numeric(length(predictors))\nnames(relevances) <- predictors\n\n## calculate, for each variate,\n## and the mutual information 'relevance' (in shannons)\n## between 'income' and that variate\nfor(var in predictors){\n relevances[var] <- mutualinfo(agent = opmall, A = 'income', B = var)\n}\n\n## output the mutual informations in decreasing order\nsort(relevances, decreasing = TRUE)\n\nmarital_status relationship education occupation workclass \n 0.10074130 0.09046621 0.06332052 0.05506897 0.03002995 \nnative_country sex race \n 0.01925227 0.01456655 0.00870089 \n\n\nIf we had to choose only one variate to infer the outcome, on average it would be best to use \\(\\mathit{marital\\_status}\\). Our last choice should be \\(\\mathit{race}\\).\n\n\n\n\n\n\n Exercise\n\n\n\nNow consider the scenario where we must exclude one variate from the eight predictors, or, equivalently, we can only use seven variates as predictors. Which variate should we exclude?\nPrepare a script similar to the one above: it calculates the mutual information between \\(\\mathit{income}\\) and the other predictors but with one omitted, omitting each of the eight in turn.\nWarning: this computation might require 10 or more minutes to complete.\n\nWhich single variate should not be omitted from the predictors? which single variate could be dropped?\nDo you obtain the same relevance ranking as in the “use-one-variate-only” scenario above?",
"crumbs": [
"[**A prototype Optimal Predictor Machine**]{.red}",
"35 [Example application: adult-income task]{.red}"
@@ -1989,7 +1989,7 @@
"href": "example_opm1.html#example-application-to-new-data",
"title": "35 Example application: adult-income task",
"section": "35.6 Example application to new data",
- "text": "35.6 Example application to new data\nLet’s apply the opmall-agent to a test dataset with 33 914 new units. For each new unit, the agent:\n\ncalculates the probability of \\(\\mathit{income}\\mathclose{}\\mathord{\\nonscript\\mkern 0mu\\textrm{\\small=}\\nonscript\\mkern 0mu}\\mathopen{}{\\small\\verb;<=50;}\\), via the function infer(), using as predictors all variates except \\(\\mathit{income}\\)\nchooses one of the two values \\(\\set{{\\small\\verb;<=50K;}, {\\small\\verb;>50K;}}\\), via the function decide() trying to maximizing utilities corresponding to the accuracy metric\n\nThe function decide() will be described in more detail in chapters 37 and 38.\nAt the end we plot a histogram of the probabilities calculated for the new units, to check for instance for how many of the agent was sure (beliefs around 0% or 100%) or unsure (beliefs around 50%). We also report the final utility/accuracy per unit, and the time needed for the computation:\n\n## Load test data\ntestdata <- read.csv('test-income_data_example.csv', header = TRUE,\n na.strings = '', stringsAsFactors = FALSE, tryLogical = FALSE)\n\nntest <- nrow(testdata) # size of test dataset\n\n## Let's time the calculation\nstopwatch <- Sys.time()\n\ntestprobs <- numeric(ntest) # prepare vector of probabilities\ntesthits <- numeric(ntest) # prepare vector of hits\nfor(i in 1:ntest){\n \n ## calculate probabilities given all variates except 'income'\n probs <- infer(agent = opmall,\n predictor = testdata[i, colnames(testdata) != 'income'])\n\n ## store the probability for <=50K\n testprobs[i] <- probs['<=50K']\n\n ## decide on one value\n chosenvalue <- decide(probs = probs)$optimal\n\n ## check if decision == true_value, and store result\n testhits[i] <- (chosenvalue == testdata[i, 'income'])\n}\n\n## Print total time required\nprint(Sys.time() - stopwatch)\n\nTime difference of 4.23584 secs\n\n## Histogram and average accuracy (rounded to one decimal)\nmyhist(testprobs, n = seq(0,1,length.out = 10), plot = TRUE,\n xlab = 'P(income = \"<=50K\")',\n ylab = 'frequency density in test set',\n main = paste0('accuracy: ', round(100*mean(testhits), 1), '%'))",
+ "text": "35.6 Example application to new data\nLet’s apply the opmall-agent to a test dataset with 33 914 new units. For each new unit, the agent:\n\ncalculates the probability of \\(\\mathit{income}\\mathclose{}\\mathord{\\nonscript\\mkern 0mu\\textrm{\\small=}\\nonscript\\mkern 0mu}\\mathopen{}{\\small\\verb;<=50;}\\), via the function infer(), using as predictors all variates except \\(\\mathit{income}\\)\nchooses one of the two values \\(\\set{{\\small\\verb;<=50K;}, {\\small\\verb;>50K;}}\\), via the function decide() trying to maximizing utilities corresponding to the accuracy metric\n\nThe function decide() will be described in more detail in chapters 37 and 38.\nAt the end we plot a histogram of the probabilities calculated for the new units, to check for instance for how many of the agent was sure (beliefs around 0% or 100%) or unsure (beliefs around 50%). We also report the final utility/accuracy per unit, and the time needed for the computation:\n\n## Load test data\ntestdata <- read.csv('test-income_data_example.csv', header = TRUE,\n na.strings = '', stringsAsFactors = FALSE, tryLogical = FALSE)\n\nntest <- nrow(testdata) # size of test dataset\n\n## Let's time the calculation\nstopwatch <- Sys.time()\n\ntestprobs <- numeric(ntest) # prepare vector of probabilities\ntesthits <- numeric(ntest) # prepare vector of hits\nfor(i in 1:ntest){\n \n ## calculate probabilities given all variates except 'income'\n probs <- infer(agent = opmall,\n predictor = testdata[i, colnames(testdata) != 'income'])\n\n ## store the probability for <=50K\n testprobs[i] <- probs['<=50K']\n\n ## decide on one value\n chosenvalue <- decide(probs = probs)$optimal\n\n ## check if decision == true_value, and store result\n testhits[i] <- (chosenvalue == testdata[i, 'income'])\n}\n\n## Print total time required\nprint(Sys.time() - stopwatch)\n\nTime difference of 4.21148 secs\n\n## Histogram and average accuracy (rounded to one decimal)\nmyhist(testprobs, n = seq(0,1,length.out = 10), plot = TRUE,\n xlab = 'P(income = \"<=50K\")',\n ylab = 'frequency density in test set',\n main = paste0('accuracy: ', round(100*mean(testhits), 1), '%'))",
"crumbs": [
"[**A prototype Optimal Predictor Machine**]{.red}",
"35 [Example application: adult-income task]{.red}"
diff --git a/example_opm1.qmd b/example_opm1.qmd
index 48413eb..2014e66 100644
--- a/example_opm1.qmd
+++ b/example_opm1.qmd
@@ -359,8 +359,7 @@ relevances <- numeric(length(predictors))
names(relevances) <- predictors
## calculate, for each variate,
-## the joint probability distribution 'probs'
-## and then mutual information 'relevance' (in shannons)
+## and the mutual information 'relevance' (in shannons)
## between 'income' and that variate
for(var in predictors){
relevances[var] <- mutualinfo(agent = opmall, A = 'income', B = var)