Subsampling rules for item non response of an estimator based on the combination of regression and ratio

2.1 Subsampling algorithm

Step 1. Select a sample s from U using simple random sampling with replacement (SRSWR).

Step 2. Evaluate Y among the subsample of the respondents $s_1\subset s$ , determine $\left\{y_i,i\in s_1,||s_1||=n_1\right\}$ and compute

Step 3. Determine $s_2=\left\{u_i\in s_2=s-s_1\right\},s_2=n_2$

Step 4. Fix $n_2^{\ast }=\theta n_2,\theta \le 1$

Step 5. Select using SRSWR a sub-sample $s_2^{\ast }{\subset s}_2\mathrm{o}\mathrm{f}\mathrm{s}\mathrm{i}\mathrm{z}\mathrm{e}{n}_2^{\ast }$

Step 6. Evaluate Y among the units in $s_2^{\ast }$ and compute

Step 7. Compute the estimate of $\stackrel{-}{Y}$

As s₁ is a subsample of U₁, (2.1) is an unbiased estimator of the mean of the response stratum, that is $E\left({\stackrel{-}{y}}_1\left|s\right.\right)={\stackrel{-}{Y}}_1$ . The subsample selected from the non-respondents provides (2.2) which is a conditionally unbiased estimator of ${\overline{y}}_2$ $\text{becaus}\text{e}E\left({\stackrel{-}{y}}_2^{\ast },\left|s\right.\right)={\stackrel{-}{y}}_2.$ Therefore, as $s_2\subset U_2,$ $E{E(\stackrel{-}{y}}_2^{\ast }\left|s\right.)={\stackrel{-}{Y}}_2$ . The usual analysis of the behavior of (2.3) is based on studying the expression

The first term is the sample mean of s, hence $\mathit{EE}(w_1{\overline{y}}_1+w_2{\overline{y}}_2)=\overline{Y},$ We have that $E\left({\overline{y}}_2^{\ast }-{\overline{y}}_2\left|s\right.\right)=0.$ Therefore, ${\stackrel{-}{y}}^{\ast }$ is an unbiased estimator of the population mean.

We have that the expected variance of the first term is

The conditional variance of the second term is given by $$V\left(w_2\left({\overline{y}}_2^{\ast }-{\overline{y}}_2\right)\left|s\right.\right)=w_2^2{\sigma }_{2Y}^2\left(\frac{1}{\theta n_2}-\frac{1}{n_2}\right)=\frac{n_2\left(1-\theta \right)}{\theta n^2}{\sigma }_{2Y}^2$$

Note that $n_2$ is a Binomial random variable, hence $$E\left[V\left(w_2\left({\overline{y}}_2^{\ast }-{\overline{y}}_2\right)\left|s\right.\right)\right]=\frac{W_2\left(1-\theta \right)}{\theta n}{\sigma }_{2Y}^2$$

Due to the independence the expectation of the cross product is zero and is deduced the well-known expression $$EV\left({\overline{y}}^{\ast }\left|s\right.\right)=\frac{{\sigma }_Y^2}{n}+\frac{W_2\left(1-\theta \right)}{\theta n}{\sigma }_{2Y}^2.$$

The value of θ determines the value of the second term in the expected error. The subsampling rules deal with determining θ. The existing particular rules fix the value of θ. They are:

• Hansen and Hurwitz (1946): $\theta =\frac{1}{K},K\ge 1$

• Srinath (1971): $\theta =\frac{n_2^2}{\mathit{Hn}+n_2},H\ge 0$

• Bouza (1981): $\theta =w_2{n}_2$

The rules of Hansen-Hurwitz and Srinath depend on the decision of the sampler for fixing θ. Bouza’s rule is determined as proportional to the proportion of non-responses. Hence, it is a randomized rule and fixing the sub-sampling size does not depend on the expertise of the sampler.

Having auxiliary information X the use of ratio estimators is commonly used. Under nonresponse we have the knowledge of the population mean of X, $\stackrel{-}{X}$ , and are computed the estimators: $${\overline{x}}^{\ast }=w_1{\overline{x}}_1+w_2{\overline{x}}_2^{\ast },\overline{x}=w_1{\overline{x}}_1+w_2{\overline{x}}_2$$ $$s_x^{\ast 2}=\frac{1}{n-1}\left(\sum _{i=1}^{n_1}{x}_i^2+\frac{n_2}{n_2^{\ast }}\sum _{j=1}^{n_2^{\ast }}{x}_j^2-n{\overline{x}}^{\ast }\overline{x}\right)$$ $$s_{\mathit{xy}}^{\ast }=\frac{1}{n-1}\left(\sum _{i=1}^{n_1}{x}_i{y}_i+\frac{n_2}{n_2^{\ast }}\sum _{j=1}^{n_2^{\ast }}{x}_j{y}_j-n{\overline{y}}^{\ast }\overline{x}\right)$$

The ratio estimator in this case is given by $${\overline{y}}_{\mathit{ratio}}^{\ast }=\frac{{\overline{y}}^{\ast }}{{\overline{x}}^{\ast }}\overline{X}.$$

The regression estimator is $${\overline{y}}_{\mathit{reg}}={\overline{y}}^{\ast }+b_{\mathit{yx}}^{\ast }\left(\overline{X}-{\overline{x}}^{\ast }\right),b_{\mathit{yx}}^{\ast }=\frac{s_{\mathit{xy}}^{\ast }}{s_x^{\ast 2}}$$

In the next section we will consider the estimation problems.

3.1 The class of estimators of Singh-Kumar

Singh and Kumar (2009) developed a class of estimators for $\stackrel{-}{Y}$ when auxiliary information on two variables X and Z is available and non-responses are present. The sampling design analyzed was a DS one. They derived expressions of the mean squared error (MSE) for the estimators of the proposed class. Take $$\overline{Q}=\sum _{j=1}^N\frac{Q_j}{N},\overline{q}=\sum _{j=1}^n\frac{q_j}{n},{\overline{q}}_t=\sum _{j=1}^{n_t}\frac{q_j}{n_t},{\overline{q}}_2^{\ast }=\sum _{j=1}^{n_2^{\ast }}\frac{q_j}{n_2^{\ast }},Q=X,Y,Z,q=x,y,z,t=1,2$$

Consider that we deal with missing information in the variable of interest (item non-response). Hence, we have responses in x and z when a unit belongs to s. Following the model, we are going to estimate the mean of Y using (2.4). We may compute $$\overline{g}=w_1{\overline{g}}_1+w_2{\overline{g}}_2,g=x,z.$$

Take α as a fixed scalar. The estimators of this class are characterized by the general formula $${\overline{y}}_{\alpha }=\left({\overline{y}}^{\ast }+{\beta }^{\ast }\left(\overline{X}-\overline{x}\right)\right)\frac{\overline{Z}}{\overline{Z}+\alpha \left(\overline{z}-\overline{z}\right)}$$ $${\beta }^{\ast }=\frac{s_{\mathit{xy}}^{\ast }}{s_x^2},s_{\mathit{xy}}^{\ast }=\sum _{i=1}^{n_1+n_2^{\ast }}\frac{\left(x_i-\overline{x}\right)\left(y_i-{\overline{y}}^{\ast }\right)}{n_1+n_2^{\ast }-1},s_x^2=\sum _{i=1}^{n_1+n_2^{\ast }}\frac{{\left(x_i-\overline{x}\right)}^2}{n_1+n_2^{\ast }-1}$$

It is considered that we know $\overline{Q},Q=X,Z$ .

Let us use a Taylor Series development for (2.4). Take $${\overline{y}}^{\ast }=\overline{y}\left(1+{\epsilon }_y\right),\overline{g}=\overline{G}\left(1+{\epsilon }_g\right),g=x,z{s}_{\mathit{xy}}^{\ast }={\sigma }_{\mathit{xy}}\left(1+{\epsilon }_{\mathit{xy}}\right),s_x^2={\sigma }_x^2\left(1+{\epsilon }_{2x}\right)$$

For accepting the validity of the development in Taylor Series is necessary that $\left|\alpha {\epsilon }_z\right|<1and\left|{\epsilon }_{2x}\right|<1\mathrm{h}\mathrm{o}\mathrm{l}\mathrm{d}$ .

From the results of Singh and Kumar (2009) we may write

where $\beta =\frac{{\sigma }_{\mathit{xy}}}{{\sigma }_x^2},R_{\mathit{xy}}=\frac{\stackrel{-}{X}}{\stackrel{-}{Y}}$ . Using the corresponding development for expanding (3.1) we have that

The approximation of the expectation, variances and covariances of the errors are developed considering that the terms of order larger than 2 are negligible. Then we may write:

where $$V_{1q}=\frac{C_q^2}{n},C_q^2=\frac{{\sigma }_q^2}{{\stackrel{-}{Q}}^2},\mathrm{q}=\mathrm{x},\mathrm{y},\mathrm{z}$$ $${V_{2qu}=\rho }_{\mathit{qu}}{C}_q{C}_u,{\rho }_{\mathit{qu}}=\frac{{\sigma }_{\mathit{qu}}}{{\sigma }_q{\sigma }_u},$$ $$V_3=\frac{N{\mu }_{21}}{\left(N-2\right)n\overline{X}{\sigma }_{\mathit{xy}}},$$ $${\mu }_{21}=\sum _{i=1}^N\frac{\left(x_i-\overline{X}\right){\left(y_i-\overline{Y}\right)}^2}{N},$$ $$V_4=\frac{N{\mu }_{30}}{\left(N-2\right)n\overline{X}{\sigma }_x^2},{\mu }_{30}=\sum _{i=1}^N\frac{{\left(x_i-\overline{X}\right)}^3}{N},$$

Let us look for the approximate bias and variance of the estimator. Considering again that the terms of order larger than 2 are negligible, we have the expansion $${\overline{y}}_{\alpha }-\overline{Y}\cong \overline{Y}\left({\epsilon }_y+\beta R_{\mathit{xy}}{\epsilon }_x\left(\alpha {\epsilon }_z-1\right)+\alpha {\epsilon }_z\left(\alpha {\epsilon }_z-{\epsilon }_y-1\right)+\beta R_{\mathit{xy}}{\epsilon }_x\left({\epsilon }_x-{\epsilon }_{\mathit{xy}}\right)\right)$$

Its expectation is equal to $$\mathit{Bias}\left({\overline{y}}_{\alpha }\right)=E\left({\overline{y}}_{\alpha }-\overline{Y}\right)\cong \overline{Y}\left(\alpha \beta R_{\mathit{xy}}{V}_{2xy}+{\alpha }^2{V}_{1z}+\alpha V_{2zy}+\beta R_{\mathit{xy}}{V}_{1x}\right)$$

Note that only ε_y is affected by the existence of missing observations. Squaring both terms and calculating the variance is obtained

The value of α determines a particular member of the class. An optimal estimator may be determined looking for the minimization of (3.5) by determining its optimum value. T is given by $${\alpha }_0=\frac{A}{R_{\mathit{yz}}}$$ which depends of unknown population parameters, see Singh and Kumar (2009) for a detailed discussion on the members of this class.

3.2 A comparison of estimators

It is well known that to the first degree of approximation of the Taylor Series the conditional variances are $$V({\overline{y}}_{\mathit{ratio}}^{\ast }\left|s)\right.\cong \frac{1}{n}\left({\sigma }_y^2+{\sigma }_x^{2}R\left(R-2B_{\mathit{yx}}\right)+\frac{n_2^{\ast }}{n}{\sigma }_{2y}^2\right)$$

For the regression estimator the conditional variance is $$V({\overline{y}}_{\mathit{reg}}\left|s)\right.\cong \frac{1}{n}\left({\sigma }_y^2\left(1-{\rho }_{\mathit{xy}}^2\right)+\frac{n_2^{\ast }}{n}{\sigma }_{2y}^2\right)$$

Noting that $$V\left({\stackrel{-}{y}}^{\ast }\left|s\right.\right)=\frac{{\sigma }_Y^2}{n}+\frac{n_2^{\ast }}{n^2}{\sigma }_{2Y}^2$$

we have that if ${\alpha }_0=\frac{A}{R_{\mathit{yz}}}$ is known $$G\left({\overline{y}}_{\mathit{ratio}}^{\ast },{\overline{y}}_{{\alpha }_0}\right)=V({\overline{y}}_{\mathit{ratio}}^{\ast }\left|s)\right.-V\left[{\overline{y}}_{{\alpha }_0}\left|s\right.\right]=\frac{1}{n}{\left(A-R_{\mathit{yz}}\right)}^2{\sigma }_z^2$$ $${G\left({\stackrel{-}{y}}_{\mathit{reg}}^{\ast },{\stackrel{-}{y}}_{{\alpha }_0}\right)=V(\stackrel{-}{y}}_{\mathit{reg}}^{\ast }\left|s)\right.-V\left[{\stackrel{-}{y}}_{{\alpha }_0}\left|s\right.\right]=\frac{A^2{\sigma }_z^2}{n}$$ $$G\left({\overline{y}}^{\ast },{\overline{y}}_{{\alpha }_0}\right)=V\left({\overline{y}}^{\ast }|s\right)-V\left[{\overline{y}}_{{\alpha }_0}\left|s\right.\right]=\frac{1}{n}\left(A^2{\sigma }_z^2+{\rho }_{\mathit{xy}}^2{\sigma }_y^2\right)$$

Hence ${\stackrel{-}{y}}_{{\alpha }_0}$ is better than the other estimators. Singh and Kumar (2011) pointed out that, even if ${\alpha }_0$ is unknown, ${\overline{y}}_{\alpha }$ is to be preferred, if the sampler evaluates for which feasible values of $\alpha$ it behaves better.

An evaluation of the magnitude of the gain in accuracy due to the use of ${\stackrel{-}{y}}_{{\alpha }_0}$ was developed using real life data.

We developed the numerical analysis using population data obtained in three studies. A brief description of them is the following

Problem 1. 793 factories contaminate a source of water. They were inspected and was obtained

• X = percent of samples with an index superior to the permitted level.

The historical report of this percent was also known

• Z = historical percent of samples with an index superior to the permitted level.

The managers improved the collection of solid contaminants and a sample an reported

• Y = percent of samples reported with an index superior to the permitted level.

N₂ = 104 factories did not send the report. They were visited and Y was obtained.

The parameters of interest are $$\overline{X}=24,7{\sigma }_x^2=31,4;\overline{Z}=10,3{\sigma }_z^2=9,34;\overline{Y}=18,7{\sigma }_y^2=7,78$$ $${\sigma }_{\mathit{yz}}=-7,96;{\sigma }_{\mathit{yx}}=10,20;{\sigma }_{\mathit{xz}}=19,62;R_{\mathit{yz}}=1,82,A=-1,19$$

Problem 2. 120 persons with VIH were included in an experiment with a new drug. The levels of hemoglobin were one of the measurements made to them. The variables involved were

• X = measurement of hemoglobin before starting with the treatment.

• Z = first measurement of hemoglobin after starting with the treatment.

• Y = measurement of hemoglobin 6 months after starting with the treatment.

N₂ = 51 patients did not visit the hospital. They were visited and Y was obtained.

The parameters of interest are $$\overline{X}=6,60{\sigma }_x^2=1,43;\overline{Z}=9,90{\sigma }_z^2=2,21;\overline{Y}=8,06{\sigma }_y^2=3,08$$ $${\sigma }_{\mathit{yz}}=0,64;{\sigma }_{\mathit{yx}}=1,01;{\sigma }_{\mathit{xz}}=0,82;R_{\mathit{yz}}=0,81,A=0,03$$

Problem 3. 1840 farmers increased the area of their farms. The interest was to evaluate the tax to be pay. The variables involved were

• X = initial area of the farms in hectares

• Z = Actual area of the farms in hectares.

• Y = Harvested area in hectares.

N₂ = 176 farmers did not return eth form of the tax to be pay patients. They were visited and Y was obtained.

The parameters of interest are $$\overline{X}=23,35{\sigma }_x^2=60,46;\overline{Z}=34,86{\sigma }_z^2=88,75;\overline{Y}=26,72{\sigma }_y^2=49,33$$ $${\sigma }_{\mathit{yz}}=15,58;{\sigma }_{\mathit{yx}}=-22,67;{\sigma }_{\mathit{xz}}=46,93;R_{\mathit{yz}}=0,77,A=0,02$$

The resulting Gains in accuracy obtained are presented in Table 1.

Table 1 suggests that the improvements in accuracy due to the use of ${\stackrel{-}{y}}_{{\alpha }_0}$ are very large when compared with ${\stackrel{-}{y}}_{\mathit{ratio}}^{\ast }$ . The error is decreased a little in the studies of VIH patients and farmers for the other estimators.

3.3 A comparison of the subsampling rules performance

From the above discussion is clear that the preference for a certain subsampling rule does not affect in the comparison of the estimators. Note that the effect of using a certain rule is important when we calculate the expected variance. Then we are interested in evaluating the behavior of the expectations under each rule. That is to compare the different expectations of $E\left[\frac{w_2\left(1-\theta \right)}{\theta n}\right]$ .

The use of Hansen-Hurwitz’s rule, HH, fixes that θ = 1/K, K ≥ 1. Then its use yields that

When we use the rule of Srinath (1971), S, we have that $$\theta =\frac{n_2}{\mathit{Hn}+n_2}$$

Doing some calculus is derived that $\frac{1-\theta }{\theta }=\frac{\mathit{Hn}}{n_2}$ . Substituting in the conditional variance, we have

Comparing this term with (3.5), we should prefer HH to S whenever $$\frac{W_2\left(K-1\right)}{n}\le \frac{H}{n}$$

That is if $K\le \frac{H+W_2}{W_2}=\frac{H}{W_2}+1.$

A similar analysis of the use of the rule of Bouza (1981) needs of considering the new structure. Due to the randomness of θ we have that the conditional variance is $$V\left(w_2\left({\overline{y}}_2^{\ast }-{\overline{y}}_2\right)\left|s\right.\right)=\frac{\left(1-n_2/n\right)}{n}{\sigma }_{2Y}^2=\frac{n_1}{n^2}{\sigma }_{2Y}^2$$

Its expectation is

Note that HH is to be preferred to B when $$\frac{W_2\left(K-1\right)}{n}\le \frac{W_1}{n}$$

That is if $K\le \frac{W_1}{W_2}+1=\frac{1}{W_2}$ as $\frac{W_1}{W_2}\ge 0$ and K ≥ 1 we may fix a value of K that satisfies this relationship.

Comparing S and B we have that the former generates a smaller coefficient if is satisfied the inequality $$H\le W_1$$

This relationship suggests that we S may be preferred if is used values of H smaller than 1.

Considering the costs, we may use the cost function $$C=c_0+c_{1}n+c_2{n}_2^{\ast }$$

Its expectation depends of the subsampling rule. The results are

Accepting that $E{\left(n_2-n_2^{\ast }\right)}^t\cong 0$ , t > 2, a development in Taylor Series allows deriving that

We have that for B

In terms of the expected costs, we may look for the preference of the rules. We have: $$\mathit{HH}\precsim SifK>H+W_2$$ $$\mathit{HH}\precsim BifK>\frac{n{W}_2}{n{W}_2^2+W_1{W}_2}$$

A comparison with S yields the preference rule, $$S\precsim \mathrm{B}\mathrm{if}\mathrm{H}>\frac{\mathrm{n}\left(1-{\mathrm{W}}_2^2\right)-{\mathrm{W}}_1}{{\mathrm{nW}}_2+{\mathrm{W}}_1{\mathrm{W}}_2}$$

It is easily derived that none of the rules may be more accurate and cheaper with respect to any of the other two simultaneously.

We used the results reported with four populations in the paper of Azeem and Hanif (2017) for establishing adequate values of the parameters of the subsampling rules. In the next table we have that $N$ is the total number of units in the population questioned, $N_1$ the number of units responding the survey questions, $N_2$ the number of units which do not respond, ${\sigma }_y^2$ is the population variance of $Y$ and ${\sigma }_y^2$ is the variance of $Y$ for non-respondents part of the population (Table 2).

Note that for the populations 1 and 2 the variance of non-respondents is similar to the overall variance. Populations 3 and 4, have a considerably larger variance of the non-respondent strata than the population variance. We will use the weights observed in the inquiries of the different non-respondent stratum, $W_2=\frac{N_2}{N}$ .

We fixed a set of values of H in Table 3 for comparing HH with S in terms of their accuracy.

Note that for H = 0,1 fixing a value of K, for which HH is to be preferred, implies using large subsample sizes. An increase in the non-respondent’s stratum determine also the need of using larger values of the sub sample size for preferring HH’s rule.

Table 4 illustrates that for small subsampling sizes HH may have a better accuracy than the rule of Bouza. S will have the same behavior for relatively small values of H.

The analysis of the costs is presented in the next 2 tables.

We prefer using HH by using $K>H+W_2$ . Analyzing Table 4 we have that in the population analyzed the relation holds for K > 1,20, which may be easily satisfied in practice.

The analysis of the costs associated with B needs to take into account the sample size. We consider the commonly used sampling fractions 0,01, 0,05 and 0,1 for illustrating. Note that the results in the Table 5 suggest that the sub sampling rule HH will have smaller expected costs than B if K > 9,82. The subsample parameter H should be very large for preferring S to B in terms of costs (Table 6).