Selecting the covariance structure for the seemingly unrelated regression models

1 Introduction

A lot of studies come across econometric models having several equations to model a real-life situation. But it so happens that the disturbance terms for such equations are somewhat correlated meaning thereby that variables affecting the disturbance term in one equation may also simultaneously affect the disturbance term in some other equation in the system of equations under study. Numerous econometrically estimated theoretical models consist of more than one equation. Overlooking such a correlation of the disturbance terms results in producing inefficient estimates of the coefficients. But estimating all such equations simultaneously with generalized least squares (GLS (estimator, considering the suitable covariance structure of the residuals, leads to efficient estimates. The nomenclature for such is commonly known as “seemingly unrelated regression equations “(SURE), Zellner (1962). Foschi and Kontoghiorghes (2003) worked on estimating regression with autoregressive disturbance. Banterle et al. (2018), Feng and Polson (2020), and Bottolo et al. (2021) worked on regression theory using the Bayesian approach.

The paper assessed methodology was given by AL-Marshadi (2014) in terms of its percentage of times that it identifies the right covariance structure for mixed model analysis of SURE models using simulated data. The simulated equations of SURE models have identical explanatory variables, the regressors in one block of equations are a subset of those in another, and different regressors in the equations with various settings of covariance structures of $\sum$ . It is expected that the proposed method will be more efficient than the existing method of considering the stander unstructured covariance structure in fitting SURE models’ since the number of the parameters in the covariance structure will be reduced when the suitable covariance structure is selected other than the stander unstructured covariance structure.

2 The SURE model

The basic model we are concerned with comprises multiple regression equations, Greene (2003).

In matrix notation, this M-equation model may be expressed more compactly as.

By writing (2) as. $${\left[\begin{array}{c}{y}_1\\ y_2\\ .\\ .\\ .\\ y_M\end{array}\right]}_{\mathit{TM}\times 1}={\left[\begin{array}{c}{X}_1\\ 0\\ \begin{array}{c}.\\ \begin{array}{c}.\\ .\end{array}\\ 0\end{array}\end{array}\begin{array}{c}0\\ X_2\\ \begin{array}{c}.\\ \begin{array}{c}.\\ .\end{array}\\ 0\end{array}\end{array}\begin{array}{c}.\\ .\\ \begin{array}{c}.\\ \begin{array}{c}.\\ .\end{array}\\ .\end{array}\end{array}\begin{array}{c}.\\ .\\ \begin{array}{c}.\\ \begin{array}{c}.\\ .\end{array}\\ .\end{array}\end{array}\begin{array}{c}.\\ .\\ \begin{array}{c}.\\ \begin{array}{c}.\\ .\end{array}\\ .\end{array}\end{array}\begin{array}{c}0\\ 0\\ \begin{array}{c}.\\ \begin{array}{c}.\\ .\end{array}\\ X_M\end{array}\end{array}\right]}_{\mathit{TM}\times K^{\ast }}{\left[\begin{array}{c}{\beta }_1\\ {\beta }_2\\ .\\ .\\ .\\ {\beta }_M\end{array}\right]}_{K^{\ast }\times 1}+{\left[\begin{array}{c}{∊}_1\\ {∊}_2\\ .\\ .\\ .\\ {∊}_M\end{array}\right]}_{\mathit{TM}\times 1}$$ the model may be expressed in the compact form.

We assume that elements of the disturbance vector ${∊}_i$ follow a multivariate probability distribution with

Consider the non-singular matrices, as $Q_{\mathit{ii}}$ and $Q_{\mathit{ij}}$ , which have the finite elements.

i.e. $\underset{T\to \infty }{Q_{\mathit{ii}}=\lim }\left(\frac{1}{T}X{\prime }_i{X}_i\right)$

and $\underset{T\to \infty }{Q_{\mathit{ij}}=\lim }\left(\frac{1}{T}X{\prime }_i{X}_j\right)$

Writing (4), (5) and (6) in compact form, we have $$E\left(\in {\in }^{\prime }\right)={\left[\begin{array}{c}{\sigma }_{11}{I}_T\\ .\\ \begin{array}{c}.\\ \begin{array}{c}.\\ .\end{array}\\ {\sigma }_{M1}{I}_T\end{array}\end{array}\begin{array}{c}.\\ .\\ \begin{array}{c}.\\ \begin{array}{c}.\\ .\end{array}\\ .\end{array}\end{array}\begin{array}{c}.\\ .\\ \begin{array}{c}.\\ \begin{array}{c}.\\ .\end{array}\\ .\end{array}\end{array}\begin{array}{c}.\\ .\\ \begin{array}{c}.\\ \begin{array}{c}.\\ .\end{array}\\ .\end{array}\end{array}\begin{array}{c}{\sigma }_{1M}{I}_T\\ .\\ \begin{array}{c}.\\ \begin{array}{c}.\\ .\end{array}\\ {\sigma }_{\mathit{MM}}{I}_T\end{array}\end{array}\right]}_{\mathit{MT}\times MT}=\left(\sum \otimes I_T\right)=\mathrm{\Psi }$$ where $\mathrm{\Psi }$ is an $(MT\times MT)$ matrix.

3 Comparison between OLS and GLS estimator

The OLS and GLS estimators are unbiased, and GLS is at least as efficient as OLS when estimating $\beta$ in the SURE model. ${\widehat{\beta }}_{\mathit{SURE}}$ is the BLUE of $\beta$ in the SURE model, since $\sum$ is non-stochastic and observable; this follows Aitken’s theorem, Aitken (1935). Looking at the expression for ${\widehat{\beta }}_{\mathit{SURE}}$ in (9), it is not an operational or feasible estimator of β because in general Σ, and hence Ψ, will be unobservable. Recognizing this, Zellner (1963) proposed an estimate of β in the SURE model, basing this on (9), but with Σ replaced by an observable M × M matrix S. In particular, the elements of S are chosen to be estimators of the corresponding elements of Σ. With this replacement for Σ, and hence $\mathrm{\Psi }$ , we now have a feasible generalized least squares FGLS estimator of β in (2):

We are assuming that the matrix S = [s_ij] is non-singular, where s_ij is some estimator of σ_ij. Although there are many possible choices of S, two ways of obtaining the s_ij are popular. Each of these is based on residuals obtained by the application of OLS in one way or another. Oberhofer and Kmenta (1974) suggested a general procedure for obtaining maximum likelihood estimates by iterating the FGLS. Direct maximum likelihood estimation can be used by inserting the special form of Σ in the log-likelihood function for the generalized regression model instead of iterated FGLS, William H. Greene, (2003). Risto and Neudecker (1997) presented the essentials of parameters estimate for the coefficients $\beta$ of the SURE model using least squares (LS), generalized least squares (GLS), and maximum likelihood (ML) (under normality). Also, parameters estimate for the variance-covariance matrix $\mathrm{\Psi }$ using an LS-related estimator and a maximum likelihood estimator (under normality) was presented with their asymptotic properties.

4 Restricted maximum likelihood estimation of the SURE model

The MIXED procedure of the SAS System has advantages over the standard multivariate procedures in fitting multiple design multivariate models (the seemingly unrelated regression models of econometrics) such as it uses observations that have incomplete responses in the calculation of the fitted models where most multivariate procedures discard an entire observation if it has any missing data. Also, it allows one to select one suitable covariance structure of the available collection covariance structures to fit the best models to data instead of restricting our analysis to unstructured covariance matrix, $\sum$ in (11), when fitting the SURE model, Wright (1998), Littell et al. (1999), and Khattree and Naik (2000). The MIXED procedure of the SAS System can be used to develop either maximum likelihood (ML) or restricted maximum likelihood (REML) estimates in order to complete the analysis of the data, where REML estimation are generally preferred to ML, Kenward and Rogers (1997). Selecting the suitable covariance structure to fit the best models to data is the concerned issue. In this paper, we suggest using AL-Marshadi (2014) method to select the suitable covariance structure to fit the best seemingly unrelated regression models to the data using the MIXED procedure with restricted maximum likelihood (REML) estimates method. Fitting the seemingly unrelated regression models with unsuitable covariance structure may impact the quality of the fitted model using the MIXED procedure, AL-Marshadi (2008). In order to evaluate the proposed suggestion, a simulation study was conducted for variety of settings of seemingly unrelated regression models such as equations that have identical explanatory variables, the regressors in one block of equations are a subset of those in another, and different regressors in the equations with various settings of covariance structures of $\sum$ . The first covariance structure was considered compound symmetry (CS) covariance structures. The second covariance structure was considered heterogeneous compound symmetry (CSH) covariance structure. The third covariance structure was considered, first-order autoregressive (AR (1)) covariance structure. The fourth covariance structure was considered heterogeneous first-order autoregressive (ARH (1)) covariance structure. The fifth covariance structure was considered, Toeplitz (TOEP) covariance structure. The sixth covariance structure was considered, Unstructured (UN) covariance structure, Littell et al. (1999). Wright (1998) explained the way of fitting the seemingly unrelated regression equations SURE models format using a MIXED procedure with MIXED Model format with restricted maximum likelihood (REML) estimates method which was used in this study.

5 The simulation study

In this section, a simulation study is carried out to assess AL-Marshadi (2014) methodology in terms of the percentage of times that it identifies the right covariance structure for mixed model analysis of SURE models’ data. Moreover, it is also reported that when the PROC MIXED procedure used REML without any intervention, the percentage of times that REML failed to converge under normal situation, Robert and Casella (2004).

From four SURE models given below, correlated multivariate normal data were generated by specifically developing SAS PROC IML code using the MIXED Model format, Wright (1998). Using two different sample sizes T = 60 and 100 with six covariance structures, resulting in 12 scenarios and for each scenario 4000 data sets were simulated. Thereafter, on each data set AL-Marshadi (2014) algorithm was employed and the percentage of times the right covariance structure is identified is reported in Table 1.

The first simulated SURE model: $$y_{t1}=3+3x_{t0}+3x_{t1}+2x_{t2}+{∊}_{t1}$$ $$y_{t2}=8+6x_{t0}+6x_{t1}+4x_{t2}+{∊}_{t2}$$ $$y_{t3}=5+9x_{t0}+9x_{t1}+6x_{t2}+{∊}_{t3}$$ $$y_{t4}=2+2x_{t0}+2x_{t1}+3x_{t2}+{∊}_{t4}$$ $$y_{t5}=-9-9x_{t0}-9x_{t1}-6x_{2t}+{∊}_{t5}$$ $$y_{t6}=-3-6x_{t0}-6x_{t1}-4x_{t2}+{∊}_{t6}$$ $$y_{t7}=-9-3x_{t0}-3x_{t1}-2x_{t2}+{∊}_{t7}$$ where $x_0$ is the dummy variable which takes the values 0 or 1, $x_1$ and $x_2$ explanatory variables following the Normal distribution with µ = 5 and σ = 2. For simulation study, correlated disturbance terms ( ${∊}_{\mathit{ti}}$ ) $\left(t=1,\cdots ,T;i=1,\cdots ,M=7\right)$ of the given equations were simulated using a different setting of the covariance matrix Ʃ.

The second simulated SURE model: $$y_{t1}=3+3x_{t1}+2x_{t2}+{∊}_{t1}$$ $$y_{t2}=8+6x_{t1}+4x_{t2}+{∊}_{t2}$$ $$y_{t3}=5+9x_{t1}+6x_{t2}+{∊}_{t3}$$ $$y_{t4}=2+2x_{t1}+3x_{t2}+{∊}_{t4}$$ $$y_{t5}=-9-9x_{t1}-6x_{2t}+{∊}_{t5}$$ $$y_{t6}=-3-6x_{t1}-4x_{t2}+{∊}_{t6}$$ $$y_{t7}=-9-3x_{t1}-2x_{t2}+{∊}_{t7}$$ where $x_1$ and $x_2$ are independent variables follow Normal distribution with µ = 5 and σ = 2. For simulation study correlated disturbance terms ( ${∊}_{\mathit{ti}}$ ) $\left(t=1,\cdots ,T;i=1,\cdots ,M=7\right)$ of the given equations were simulated using the different settings of the covariance matrix Ʃ.

The third simulated SURE model: $$y_{t1}=3+{∊}_{t1}$$ $$y_{t2}=9+6x_{t0}+3x_{t1}+{∊}_{t2}$$ $$y_{t3}=7+5x_{t1}+{∊}_{t3}$$ $$y_{t4}=12+2x_{t2}+{∊}_{t4}$$ $$y_{t5}=5+22x_{t1}+2x_{2t}+{∊}_{t5}$$ $$y_{t6}=1+18x_{t1}+52x_{t2}+{∊}_{t6}$$ $$y_{t7}=5+12x_{t0}+2x_{t1}+2x_{t2}+{∊}_{t7}$$ where $x_0$ is a dummy variable taking values 0 or 1, $x_1$ and $x_2$ explanatory variables following normal distribution with µ = 5 and σ = 2. For simulation study, correlated disturbance terms ( ${∊}_{\mathit{ti}}$ ) $\left(t=1,\cdots ,T;i=1,\cdots ,M=7\right)$ of the given equations were simulated using a different setting of the covariance matrix Ʃ.

The fourth simulated SURE model: $$y_{t1}=3+{∊}_{t1}$$ $$y_{t2}=8+3x_{t1}+{∊}_{t2}$$ $$y_{t3}=7+5x_{t1}+{∊}_{t3}$$ $$y_{t4}=12+2x_{t2}+{∊}_{t4}$$ $$y_{t5}=5+22x_{t1}+2x_{2t}+{∊}_{t5}$$ $$y_{t6}=1+18x_{t1}+52x_{t2}+{∊}_{t6}$$ $$y_{t7}=12+2x_{t1}+2x_{t2}+{∊}_{t7}$$ where $x_1$ and $x_2$ are explanatory variables following the normal distribution with µ = 5 and σ = 2. For simulation study correlated disturbance terms ( ${∊}_{\mathit{ti}}$ ) $\left(t=1,\cdots ,T;i=1,\cdots ,M=7\right)$ of the given equations were simulated using a different setting of the covariance matrix Ʃ.

6 Results

Table 2 exhibits summarized results of AL-Marshadi (2014) approach selecting the right covariance structure percentage of times from the six Covariance structures, when W = 10, and T = 60, where ‘W’ represents the number of the bootstrap samples. Table 3 exhibits summarized results of AL-Marshadi (2014) approach selecting the right covariance structure percentage of times from the six Covariance structures, W = 10, and T = 100, where ‘W’ represents the number of the bootstrap samples and this is in line with the suggestions provided in AL-Marshadi (2014). Results of Table 2 and Table 3 show similar reliable performance with a high percent of success in selecting the suitable covariance structure with AL-Marshadi (2014) approach across all the simulated SURE models with the different covariance structures which will be allowed the SURE models fitted more efficient than the existing method of considering the stander unstructured covariance structure in fitting SURE models’ since the number of the parameters in the suitable selected covariance structure will be reduced when the suitable covariance structure is selected other than the stander unstructured covariance structure. The comparison of the efficiency of fitting the SURE models using the stander unstructured covariance structure or using the suitable selected covariance structure with AL-Marshadi (2014) approach will be clear in the application using a panel of data. Results of Tables 2 and 3 also suggest that the performance of the approach has enhanced with the increase in the sample size with constant bootstrap samples.

Tables 4 and 5 exhibit the percentage of times that the PROC MIXED procedure failed to converge when the PROC MIXED procedure used REML without any interfering for all the investigated settings of the covariance matrix and W = 10, and T = 60 and W = 10, and T = 100. To put it in a nutshell, the results in Tables 4 and 5 suggest that increasing the sample size addresses the convergence problem (Table 6).

7 Application using grunfeld’s data

To illustrate the AL-Marshadi (2014) approach to select the right covariance structure among the seven covariance structure considered in the study using a panel of data that serve as a useful tool for investigating multiple equation estimators for a long period of time in the literature. The data consist of time series of twenty yearly observations for five firms and three variables:

$I_t$  = gross investment.

$F_t$  = market value of the firm at the end of the previous year.

$C_t$  = value of the stock of plant and equipment at the end of the previous year, William H. Greene, (2003), and Grunfeld (1958). The estimated model with these data is. $$I_{\mathit{it}}={\beta }_{1i}+{\beta }_{2i}{F}_{\mathit{it}}+{\beta }_{3i}{C}_{\mathit{it}}+{∊}_{\mathit{it}}$$ where i indexes firms and t indexes years.

Therefore, the AL-Marshadi (2014) approach was applied to select the best covariance structure among the seven covariance structure considered in the study for the data as follow:

1. We generated the bootstrap samples on a case-by-case using the original data (i.e., based on resampling from twenty-year observations). The bootstrap sample size is taken to be the same as the size of the original data (i.e., 20 years).

2. The model was fitted with the candidate covariance structures that we like to select the best covariance structure from them, to each of the bootstrap samples, thereby obtaining the bootstrap

${\mathit{AIC}}^{\ast }{\mathit{BIC}}^{\ast }{\mathit{CAIC}}^{\ast }{\mathit{HQIC}}^{\ast }{\mathit{AICC}}^{\ast }$ for the model with the candidate covariance structures.

1. Repeat steps (1) and (2) (10) times

2. Bootstrapping of the original data is given us the opportunity to have (10) replication values for each model and each information criteria (from steps (1to 3). we used the average of each information criteria for each model sepyarately in the algorithm as a random vector that follows 5-dimensional multivariate normal distribution: ${\left[\overline{\mathit{AIC}}\overline{\mathit{BIC}}\overline{\mathit{CAIC}}\overline{\mathit{HQIC}}\overline{\mathit{AICC}}\right]}_{\mathit{Coveriancstructure}-i}$ .

In this stage, clustering method was used to cluster the candidate covariance structures s two clusters according to the five correlated variables (the averages of the five information criteria). One of the two clusters is called the cluster of the best set of covariance structures according to the cluster that contains the general covariance structure UN (Unstructured covariance structure). Then the best covariance structure is the simplest covariance structure in the cluster of the best set of covariance structures. The results for the data are given in Table 7 that suggest CSH is the best covariance structure for the data. The efficiency of using the best suitable covariance structure for the data instead of using the stander unstructured covariance structure for the data is shown in Table 7 which compares the stander error for estimated parameters for the two covariance structures. The comparison of the stander error for estimated parameters in Table 7 shows the stander error for estimated parameters with suitable selected covariance structure with AL-Marshadi (2014) approach are less than the stander error for estimated parameters with the stander unstructured covariance structure.

8 Conclusions

In our simulation, we considered SURE models, examining the performance of the AL-Marshadi (2014) approach to select the right covariance structure with different covariance structure settings. Overall, the AL-Marshadi (2014) approach provided an excellent tool for selecting the right covariance structure for SURE models in order to fit the SURE models more efficiently than the existing method that considering the stander unstructured covariance structure in fitting SURE models that can be seen with the application of the panel of data when the standard errors compared in Table 7. Future work can be considered to examine the performance of the AL-Marshadi (2014) approach under the existent of multicollinearity.