Post on 22-Feb-2016
description
transcript
SJTU CMGPD 2012Methodological Lecture
Day 2
TABLE, COLLAPSE, HISTOGRAM, TWOWAY BAR
Descriptive statistics• There are a number of ways in STATA of transforming the
dataset to produce descriptive statistics to be plotted or put into a figure
• Slow, manual way– TABULATE– Copy results to Excel, parse, and plot– Not recommended
• Transformation to produce counts, averages etc. according to the values of specified variables to use as the basis of plots– TABLE, REPLACE– COLLAPSE– BYSORT combined with EGEN (to be discussed later)
Collapsing the data
• TABLE, REPLACE and COLLAPSE transform the data
• For each value of a specified variable, or each combination of values for specified variables, produce a single observation with summary statistics of other specified values
• These summary statistics can be counts, sums, means, etc.
COLLAPSE Start with a hypothetical dataset
+----------------+ | x1 x2 y | |----------------| 1. | 1 3 12 | 2. | 2 3 100 | 3. | 1 3 45 | 4. | 2 3 -18 | 5. | 1 3 73 | |----------------| 6. | 2 4 22 | 7. | 1 4 -129 | 8. | 2 4 -100 | 9. | 1 4 -9 | 10. | 2 4 112 | +----------------+
Replace the dataset with one that for each combination of x1 and x2, contains the mean of y
. collapse y, by(x1 x2)
. list
+--------------------+ | x1 x2 y | |--------------------| 1. | 1 3 43.33333 | 2. | 1 4 -69 | 3. | 2 3 41 | 4. | 2 4 11.33333 | +--------------------+
Or count the numbers of records for each unique combination of x1 and x2
. collapse (count) y, by(x1 x2)
. list
+-------------+ | x1 x2 y | |-------------| 1. | 1 3 3 | 2. | 1 4 2 | 3. | 2 3 2 | 4. | 2 4 3 | +-------------+
Or both at the same time, creating count and average simultaneously. ‘avgy=‘ tells it to create a new variable name.
. collapse (count) y (mean) avgy=y, by(x1 x2)
. list
+------------------------+ | x1 x2 y avgy | |------------------------| 1. | 1 3 3 43.33333 | 2. | 1 4 2 -69 | 3. | 2 3 2 41 | 4. | 2 4 3 11.33333 | +------------------------+
TABLE, REPLACECan achieve the same thing with TABLE, REPLACE, though the resulting variable names are a bit cryptic
. table x1 x2, contents(count y mean y) replace
------------------------------ | x2 x1 | 3 4----------+------------------- 1 | 3 2 | 43.33333 -69 | 2 | 2 3 | 41 11.33333------------------------------
. list
+-----------------------------+ | x1 x2 table1 table2 | |-----------------------------| 1. | 1 3 3 43.33333 | 2. | 1 4 2 -69 | 3. | 2 3 2 41 | 4. | 2 4 3 11.33333 | +-----------------------------+
.
histogramObservations by year
The easy way to get a figure for numbers of observations by register year is to use histogram.
histogram YEAR, discrete frequency ytitle("Observations") xtitle("Year") xlabel(1750(25)1900)
To force a monochromatic color scheme, we can add scheme(s1mono)
To override the default numeric format of the vertical axis labels, we can add ylabel(,format(“%5.0f”))
histogram YEAR, discrete frequency ytitle("Observations") xtitle("Year") xlabel(1750(25)1900) ylabel(,format(%5.0f)) scheme(s1mono)
02.
0e+0
44.
0e+0
46.
0e+0
48.
0e+0
4O
bser
vatio
ns
1750 1775 1800 1825 1850 1875 1900Year
020
000
4000
060
000
8000
0O
bser
vatio
ns
1750 1775 1800 1825 1850 1875 1900Year
histogramRestricting the data
• Often, in producing a histogram, it is necessary to prevent the display of invalid, implausible, or otherwise problematic observations.– Missing values are always coded as -98 or -99, and should be
excluded from graphs• Do this with an if restriction in the command• This applies to tables as well.• Compare the results of
– histogram AGE_IN_SUI– histogram AGE_IN_SUI if AGE_IN_SUI >=1 & AGE_IN_SUI <= 99
if and logical expressions in STATA• if AGE_IN_SUI >=1 & AGE_IN_SUI <= 99 restricts the
command to observations where AGE_IN_SUI is >=1, and <= 99.• & represents AND
– Expression is evaluated as true only if ALL expressions are TRUE • | represents OR
– Expression is evaluated as true if ANY of the expressions are TRUE• May use parentheses (, ) to specify order of evaluation• ! represents NOT
– In a logical expression, TRUE is typically indicated as 1, and FALSE is indicated as 0.• If AGE_IN_SUI was 45, AGE_IN_SUI >= 1 would evaluate to 1, and
AGE_IN_SUI <= 99 would evaluate to 1.– 1 & 1 would evaluate to 1, TRUE
• If AGE_IN_SUI was 105, AGE_IN_SUI >= 1 would evaluate to 1, and AGE_IN_SUI <= 99 would evaluate to FALSE or 0
– 1 & 0 would evaluate to 0, FALSE
histogramSome additional options
• Tell STATA that the values are discrete, not continuous:– histogram AGE_IN_SUI if AGE_IN_SUI >= 1 & AGE_IN_SUI <= 99,
discrete• Set the Y-axis to represent percentages:
– histogram AGE_IN_SUI if AGE_IN_SUI >= 1 & AGE_IN_SUI <= 99, percent discrete
• Customize labeling of the X-axis– histogram AGE_IN_SUI if AGE_IN_SUI >= 1 & AGE_IN_SUI <= 99,
percent discrete xlabel(0(10)100)• Add tick marks to the X axis
– histogram AGE_IN_SUI if AGE_IN_SUI >= 1 & AGE_IN_SUI <= 99, percent discrete xlabel(0(10)100) xtick(0(5)100)
• Produce separate graphs according to the value of another variable– histogram AGE_IN_SUI if AGE_IN_SUI >= 1 & AGE_IN_SUI <= 99 &
(SEX != -99), percent discrete xlabel(0(10)100) xtick(0(5)100) by(SEX)
table and bar to produce histogramsObservations by year
We could do the same thing with table to prepare the dataset, and then twoway bar.
table YEAR, contents(freq) replacetwoway bar table1 YEAR, scheme(s1mono) xlabel(1750(25)1900) ytitle("Number of observations")
Or if we want to do it as a scatter plot…twoway scatter table1 YEAR, scheme(s1mono) xlabel(1750(25)1900) ytitle("Number of observations")
020
,000
40,0
0060
,000
80,0
00N
umbe
r of o
bser
vatio
ns
1750 1775 1800 1825 1850 1875 1900Year
020
,000
40,0
0060
,000
80,0
00N
umbe
r of o
bser
vatio
ns
1750 1775 1800 1825 1850 1875 1900Year
Registers by year
• The number of available registers varies year by year.
• This accounts for some of the year to year fluctuation in numbers of observations
• In some cases, may also account for some of the year to year fluctuation in other summary values
• We can do a year by year count of the number of available registers easily enough
Registers by yeartable YEAR DATASET, replacetable YEAR, replacetwoway bar table1 YEAR, scheme(s1mono)
ytitle("Registers")Let’s use angle and labsize on xlabel to label each register year
individuallytwoway bar table1 YEAR, scheme(s1mono)
ytitle("Registers") xlabel(1750(3)1909,angle(vertical) labsize(vsmall))
• Note that coverage is much more sparse before 1789.• Some years (1810) are missing an especially large number of registers• No registers at all from 1888 to 1903
010
2030
Reg
iste
rs
1750
1759
1768
1777
1786
1795
1804
1813
1822
1831
1840
1849
1858
1867
1876
1885
1894
1903
1912
Year
010
2030
Reg
iste
rs
1750
1753
1756
1759
1762
1765
1768
1771
1774
1777
1780
1783
1786
1789
1792
1795
1798
1801
1804
1807
1810
1813
1816
1819
1822
1825
1828
1831
1834
1837
1840
1843
1846
1849
1852
1855
1858
1861
1864
1867
1870
1873
1876
1879
1882
1885
1888
1891
1894
1897
1900
1903
1906
1909
Year
Population by age groupLet’s use TABLE to look at the distribution of the population by age group
keep if PRESENT & AGE >= 1 & AGE <= 75, clearrecode AGE_IN_SUI 1/15=1 16/55=16 56/75=56,
generate(AGE_GROUP)tab AGE_GROUP SEX if SEX >= 1, col rowtable AGE_GROUP SEX if SEX >= 1, col row
recode maps values of an existing variable to new values, based on the specified rule. If generate is not specified, it transforms the existing variables. If generate is specified, it creates a new variable with the new values. In this case, all AGE_IN_SUI 1 through 15 all get converted to 1, 16 through 55 are converted to 16, and so forth.
RECODE of |AGE_IN_SUI | (Age in | Sex Sui) | Female Male | Total-----------+----------------------+---------- 1 | 36,300 234,332 | 270,632 | 13.41 86.59 | 100.00 | 6.88 28.12 | 19.88 -----------+----------------------+---------- 16 | 393,977 500,381 | 894,358 | 44.05 55.95 | 100.00 | 74.67 60.04 | 65.71 -----------+----------------------+---------- 56 | 97,333 98,716 | 196,049 | 49.65 50.35 | 100.00 | 18.45 11.84 | 14.40 -----------+----------------------+---------- Total | 527,610 833,429 | 1,361,039 | 38.77 61.23 | 100.00 | 100.00 100.00 | 100.00
-------------------------------------RECODE of |AGE_IN_SU |I (Age in | Sex Sui) | Female Male Total----------+-------------------------- 1 | 36,300 234,332 270,632 16 | 393,977 500,381 894,358 56 | 97,333 98,716 196,049 | Total | 527,610 833,429 1361039-------------------------------------
Counts, averages, proportions by age and time
• There are a variety of options for collapsing observations to produce counts, proportions, averages, etc. by year, age, etc.
• One simple approach is the table command, combined with the replace option
• This replaces the dataset in memory with a ‘collapsed’ version
• Values in the ‘collapsed’ version can be plotted with twoway bar etc.
table AGE_GROUP SEX if SEX >= 1, by(YEAR) replace* Entries created for totals have missing values for AGE_GROUPdrop if AGE_GROUP == .reshape wide table1, i(YEAR SEX) j(AGE_GROUP)* Also need to remove newly created totals with missing values for SEXdrop if SEX == .reshape wide table11 table116 table156, i(YEAR) j(SEX)generate male_proportion_16_55 = table1162/(table112+table1162+table1562)twoway bar male_proportion_16_55 YEAR, ytitle("Proportion of males who
are 16 to 55 sui") xtitle("Year") ylabel(0(0.1)1) scheme(s1mono)generate male_dependency_ratio = (table112+table1562)/(table1162)twoway bar male_dependency_ratio YEAR, ytitle("Male dependency ratio ((1-
15 + 56-75)/(16-55) ") xtitle("Year") ylabel(0(0.1)1) scheme(s1mono)generate child_sex_ratio = table112/table111twoway bar child_sex_ratio YEAR, ytitle("Ratio of males to females aged
1-15 sui") xtitle("Year") scheme(s1mono) yscale(log) ylabel(1 2 5 10 20 50 100 200)
Reshape• Notice that TABLE (and COLLAPSE) will produce one observation
for each combination of YEAR, age_group, and SEX• 50*3*2=300 observations (approximately)
– 299 in reality because one cell is empty• We would like one observation per year
– In order to carry out calculations• Use reshape to convert to one observation per combination of
YEAR and SEX, with three variables, one each for each of the age groups
• Use reshape again to convert to one observation per YEAR, with six variables per observation, one for each combination of SEX and age_group
• Can calculate dependency ratios, sex ratios etc. from these numbers
0.1
.2.3
.4.5
.6.7
.8.9
1P
ropo
rtion
of m
ales
who
are
16
to 5
5 su
i
1750 1800 1850 1900Year
0.1
.2.3
.4.5
.6.7
.8.9
1M
ale
depe
nden
cy ra
tio ((
1-15
+ 5
6-75
)/(16
-55)
1750 1800 1850 1900Year
12
510
2050
100
200
Rat
io o
f mal
es to
fem
ales
age
d 1-
15 s
ui
1750 1800 1850 1900Year
Proportions/meansProportion ever married by year
We can also calculate means of specified variables by YEAR, AGE_IN_SUI, or other variables of interest
use "C:\Users\Cameron Campbe\Documents\Baqi\CMGPD-LN from ICPSR\ICPSR_27063\DS0001\27063-0001-Data.dta" if PRESENT & AGE >= 16 & AGE <= 50 & SEX == 2 & MARITAL_STATUS >= 0, clear
recode AGE_IN_SUI 16/30=16 31/40=31 41/50=41, generate(age_group)generate ever_married = MARITAL_STATUS != 2table YEAR age_group, contents(mean ever_married) replacetwoway bar table1 YEAR if age_group == 16,ylabel(0(0.1)1)
ytitle("Proportion of men 16-30 ever married") xtitle("Year") scheme(s1mono)
twoway bar table1 YEAR if age_group == 31,ylabel(0(0.1)1) ytitle("Proportion of men 31-40 ever married") xtitle("Year") scheme(s1mono)
0.1
.2.3
.4.5
.6.7
.8.9
1P
ropo
rtion
of m
en 1
6-30
eve
r mar
ried
1750 1800 1850 1900Year
0.1
.2.3
.4.5
.6.7
.8.9
1P
ropo
rtion
of m
en 3
1-40
eve
r mar
ried
1750 1800 1850 1900Year
Proportion married by ageuse "C:\Users\Cameron Campbe\Documents\Baqi\CMGPD-LN from
ICPSR\ICPSR_27063\DS0001\27063-0001-Data.dta" if PRESENT & AGE >= 1 & AGE <= 50 & SEX == 2 & MARITAL_STATUS >= 0, clear
generate ever_married = MARITAL_STATUS != 2table AGE_IN_SUI, contents(mean ever_married) replacetwoway bar table1 AGE_IN_SUI, ylabel(0(0.10)1)
ytitle("Proportion of males ever married") xtitle("Age in sui") scheme(s1mono)
0.1
.2.3
.4.5
.6.7
.8.9
1P
ropo
rtion
of m
ales
eve
r mar
ried
0 10 20 30 40 50Age in sui
Multiple trends in the same graphkeep if SEX == 2 & PRESENT & BIRTHYEAR >= 1750 & BIRTHYEAR <=
1900keep if MARITAL_STATUS > 0keep if AGE_IN_SUI >= 11 & AGE_IN_SUI <= 40recode AGE_IN_SUI 11/15=11 16/20=16 21/25=21 26/30=26 31/35=31
36/40=36, generate(age_group)generate ever_married = MARITAL_STATUS != 2table BIRTHYEAR age_group, contents(mean ever_married) replacetwoway line table1 BIRTHYEAR if age_group == 11 || line table1
BIRTHYEAR if age_group == 16 || line table1 BIRTHYEAR if age_group == 21 || line table1 BIRTHYEAR if age_group == 26 || line table1 BIRTHYEAR if age_group == 31 || line table1 BIRTHYEAR if age_group == 36 || ,scheme(s1mono) legend(order(1 "11-15 sui" 2 "16-20 sui" 3 "21-25 sui" 4 "26-30 sui" 5 "31-35 sui" 6 "36-40 sui")) ytitle("Proportion of males ever married")
0.2
.4.6
.81
Pro
porti
on o
f mal
es e
ver m
arrie
d
1750 1800 1850 1900Year of Birth
11-15 sui 16-20 sui21-25 sui 26-30 sui31-35 sui 36-40 sui
Using COLLAPSEkeep if PRESENT & SEX == 2 & AGE_IN_SUI > 1 & AGE_IN_SUI
<= 60mvdecode _all, mv(-99 -98)generate MARRIED = MARITAL_STATUS == 1
By default, collapse will create variables of the same name containing means
collapse MARRIED SON_COUNT DAUGHTER_COUNT FATHER_ALIVE MOTHER_ALIVE BROTHER_COUNT, by(AGE_IN_SUI)
Notice use of legend to specify a label for each of the 5 linestwoway line FATHER_ALIVE MOTHER_ALIVE MARRIED SON_COUNT
BROTHER_COUNT AGE_IN_SUI, scheme(s1mono) legend(order(1 "Father alive" 2 "Mother alive" 3 "Wife alive" 4 "Sons ever born" 5 "Brothers alive")) ytitle("Mean") lpattern(solid solid dash dot dash_dot)
0.5
11.
5M
ean
0 20 40 60Age in Sui
Father alive Mother aliveWife alive Sons ever bornBrothers alive
Calculating rates• Calculation of demographic rates by age and so forth is
straightforward, using the AT_RISK_* and NEXT_* flag variables.• Let’s calculate and compare probability of marriage in the next
three years by age, for men and women
keep if AT_RISK_MARRY == 1 & SEX > 0 & AGE_IN_SUI > 0 & AGE_IN_SUI <= 30
collapse NEXT_MARRY, by(AGE_IN_SUI SEX)twoway line NEXT_MARRY AGE_IN_SUI if SEX == 1 || line NEXT_MARRY AGE_IN_SUI if SEX == 2 || , legend(order(1 "Female" 2 "Male")) scheme(s1mono)
0.2
.4.6
Pro
porti
on m
arry
ing
in n
ext 3
yea
rs
1 6 11 16 21 26 31Age in Sui
Female Male