Copyright Wolfgang M. Hartmann. Last revised: October 2018 List of Content Offering Consulting Service Dates of CMAT Releases Wolfgang's Coefficient of Usability of a Software User Software License Agreement Warranty Installation of CMAT Running CMAT Some known Problems with CMAT Data Types in CMAT Technical Papers CMAT Newsletters Things ToDo New Developments Since Release 3 in Dec. 2002 "Algorithm" by Paul Boggs

Since September 28, 2016, there is a new version of CMAT ready for download from the web. This is probably the most bug and error free version of the last five releases. There are maybe still a very few minor things I could fix, maybe at the end of this year 2016. Due to another and more urgent project, I must have a break in coding CMAT starting Oktober 2016. For a 73 year old it will not be easy to continue the CMAT coding in summer of 2017.

Extreme Voting Results in Germany

Offering Consulting Service

• Structural Equation Modeling Some Guidelines
  • with our automatic modeling function we are creating excellent CFA (confirmatory factor analysis) models for your data;
    (at no charge if we are not able to find models with p larger than 0.01)
  • structural equation modeling (SEM) for metric and ordinal data (multiple sample analysis)
  CFA model improvement algorithm
• Linear and Nonlinear Optimization: using CMAT and Matlab (Optimization Toolbox)
• Linear and Nonlinear Statistics:
  • dimension reduction and variable selection
  • normalization and analysis of microarray data (using Bioconductor)
  • structural equation modeling (SEM), IRT, factor analysis, and PLS
• Data Mining with CMAT, R, and SAS Enterprise Miner (v. 9.1.3)
• Programming in SAS: DATA Step, IML, STAT, ETS, OR, and EM (using SAS software version 9.1.3)
• Programming CMAT, Matlab, R, C, and Fortran

Dates of CMAT Releases

1. Release 1: 1996 (Copyright February 1997)
2. Release 2: December 1999
3. Release 3: December 2002
4. Release 4: July 2007
5. Release 5: January 2009
6. Release 6: November 2011: I'm now developing with MS Visual Studio 2010 (C/C++) and Intel Parallel Studio (Fortran 90)
7. Release 7: December 2013
8. Release 8: December 2015
9. Release 9: September 2016

1. Somebody who does some testing of the language and functions. Enters bug reports. Would need to have some Math and Stat background.
2. Somebody who does some marketing. Would need to know about competing software in Math and Stat (Octave, Matlab, and SAS IML).

WCU: Wolfgang's Coefficient of Usability of Software

Wolfgang defines WCU as the ratio among the number of people who actually use a software (i.e. would be even willing to pay for the use) and the number of people who developed that software. Now, since CMAT has only one developer, the smallest numbers of WCU are 0 and 1, and compared to the high values of WCU for SPSS and SAS (not even thinking of Google and Facebook) CMAT seems to be rather bad. However, quite a number of packages of R don't do much better than CMAT:-)

User Software License Agreement

1. This software may not be distributed to third parties. The free version may only be used for non-profit research and teaching.
2. Using the software for commercial applications or for profit needs a specific license agreement.
3. Supply of any part of this software as part of another software requires the separate prior written agreement of the CMAT managers, which may include financial terms.
4. This software is copyrighted and may not be modified, decompiled, reverse engineered, or disassembled.
5. Due acknowledgment shall be made of the use of CMAT in research reports or publications.

Warranty

On the Installation of CMAT: Last upload date: September 28, 2016

The following directory structure is highly recommended:

1. Download and unzip the file (about 6 MB): cmat_util.zip creating the directory C:/cmat_util containing the directory gnuplot and the files for using 7z which is used by the CMAT function system().
2. Create directory C:/cmat.
   1. Download into cmat and unzip the file (about 16 MB) cmat_com.zip creating the subdirectory cmat/com which contains the executable cmat.exe, a number of DLLs, and the message files for the output.
   2. Download into cmat and unzip the file (about 62 MB) cmat_test.zip creating the following subdirectories:
      • cmat/test with general test examples
      • cmat/tgplt with test examples for plotting
      • cmat/tmicro with test examples for micro array data
      • cmat/tnlp with test examples for optimization (LP and NLP)
      • cmat/tsem with test examples for structural equation modeling, factor analysis, rotation, and IRT modeling
   3. Download into cmat and unzip the file (about 24 MB) cmat_data.zip creating the directory cmat/data which contains the following subdirectories:
      • mytst is almost empty and would be some good place for users work
      • tdata contains a large number of data sets, many of them used from the test examples. All files have the extension .dat
   4. Download into cmat and unzip the file (about 430 MB!) cmat_save.zip creating the cmat/save which contains a number of .dob files generated using the obj2fil() function which can be read by the fil2obj() function. These files are used by some applications with very large data sets. This directory is only needed if you use the functions obj2fil() and fil2obj() in your CMAT input.
   5. Download into cmat and unzip the file (about 14 MB) cmat_doc.zip creating the directory cmat/doc which contains a number of pdf files documenting CMAT, including all Newsletters (see also below). The files for this directory can also be accessed directly on the wcmat website.

When e.g. EMACS is installed, CMAT can now be used in command line or batch mode in Windows, preferably in C:/cmat/mytst.

Only when using the Tcl/TK graphical interface, for online documentation Adobe Reader must be installed in a directory Reader (containing files AcroRd32.exe, AcroRd32.dll etc.) preferably at the same hierarchy level as the cmat directory. Then the help("string") function can be used to open the reference manual at the specified term. You may move easily between the more than 400 terms (bookmarks) of the CMAT Reference Manual by clicking the many more hyperlinks. The manual can also be opened at a specific page or for searching a specified term. The Tcl/TK GUI can be used also for accessing the reference manual.

For graphical output (plotting) CMAT has an interface to the gnuplot software. In CMAT you may connect to gnuplot either interactively or in batch mode (running scripts). Most Linux distributions have gnuplot included. For Windows and other OS gnuplot can be downloaded from the internet free of charge, see for example here (There is also a demo gallery.) For some "terminal" output of gnuplot (like SVG, EPSLATEX, and PDF) you may have to download some additional software (SVGViewer, graphixs, etc). An excellent book about gnuplot is: Philip K. Janert, "Gnuplot in Action", Greenwich CT: Manning Publications Co., 2009 Toshihiko Kawano has his "not-so-freqently asked questions" here and here something about gnuplot tricks.

On Running CMAT

1. with one of the two available graphical interfaces, one in Windows and one in Unix style,
2. or either by calling cmat.exe in a DOS Command window, or using Emacs (and evtl. BASH) in Unix style. See remarks in the CMAT Tutorial document.

It is best to run CMAT (either interactively or in Batch mode) from the mytst, test, tnlp, or tsem directory. The test, tnlp, and tsem directories contain a large number of batch example files all ending with the extension .inp together with the .log and .txt output files. When running those examples, e.g.

cmat tode.inp

you should obtain the same results, but the log file will show the actual date of the execution. Note, that the CMAT input must always be started with a { bracelet for the start of a compound statement. At least one statement (maybe an empty one) must be run before the script is closed by a mstching } bracelet.

You will find more information about downloading and running CMAT in Windows, Linux, and Unix at the download site.

Some known Problems with CMAT

• There are some problems with the import() and export() functions. They will probably be fixed during the next few months.
• There is a problem with the con (LPasL1) version of the lp() function. The pcx and clp versions can be used instead.

Selected CMAT Documents

1. Complete User's Manual (about 18 mb, close to 3000 pages):
   1. User Software License Agreement
   2. Introduction
   3. Installing and Running CMAT
   4. Restrictions and Comparisons
   5. Tutorial: Basic Elements of the CMAT Language
   6. Summary of Operators, Keywords, and Functions
   7. Reference Guide
   8. Some Details
   9. The Bibliography

2. CMAT Reference Manual(about 14 mb, more than 2000 pages):
   1. Reference Guide
   2. The Bibliography

3. CMAT Tutorial (about 200 pages):
   1. Introduction
   2. Tutorial: Basic Elements of the CMAT Language
   3. Summary of Operators, Keywords, and Functions
   4. The Bibliography

4. CMAT Details and Examples:
   1. Details
   2. The Bibliography

5. CMAT Summary Manual (about 110 pages):
   1. Introduction
   2. Summary of Operators, Keywords, and Functions
   3. The Bibliography
   Here some short guidelines about how to use the hyperref package in LaTeX.

   ---

   Newsletters

   Please note that the developments reported in the last posted newsletter must not be implemented in the last posted software version on the net or are not much tested and may not work with the software posted at the site.

   January 2003 March 2003 May 2003 July 2003 September 2003 November 2003 January 2004 June 2004 December 2004

   July 2005 December 2005 July 2006 December 2006 July 2007 December 2007 July 2008, Part I July 2008, Part II

   December 2008 July 2009 December 2009 July 2010 December 2010 July 2011 December 2011 July 2012

   July 2013 December 2013 July 2014 December 2014 December 2015 July 2016 July 2017 July 2018

   ---

   Data Types in CMAT

   1. Scalars: (long) int, (double) real, (double) complex, string
   2. Vectors (dense and sparse) for all data types, even mixed
   3. Matrices (dense and sparse) for all data types, even mixed and for some specific matrix types (diagonal, band, symmetric, triangular)
   4. Tensors (dense and sparse) for all data types, even mixed
   5. Lists, where each entry can be scalar, vector, matrix, tensor, list, or struct; entries are referred to by index
   6. Structures (since end of 2015), where each entry can be scalar, vector, matrix, tensor, list, or struct; entries are referred to by compound name
   7. KD Trees (not completely finished, internally only)
   1. An Important Language Extension:
      This two page paper describes a new form of matrix literal permitting the input of matrices containing string data without quotes. (Note, for using that unquoted form of string input, matrix literals may not contain white space inside.)
   2. Data Objects in CMAT:
      This paper sketches some aspects of the data objects implemented in CMAT.
   3. Tensor and List Operations in CMAT:
      Many matrix operations have been extended to tensors. However, this paper sketches also some more specific and additional operations for tensors and data lists.

   Some Technical Papers

   The download site also contains a number of technical reports illustrating applications of CMAT:
   1. Semiannual Newsletters Starting 2003
      The CMAT Newsletters report about the progress in the development and illustrate some applications of CMAT.
   2. On the Use of Matrix Language:
      This small paper illustrates the difference between educational and efficient programming. Note, CMAT almost always knows when a matrix is symmetric and takes advantage of this. Also, identity matrices are stored as diagonal matrices. Sparsity in matrices and vectors is detected automatically. Such examples are often found in statistics.
   3. CMAT Code for some Matlab Programs by
      Olvi Mangasarian and Helen Zhang.
   4. Presentation at DAGStat Conference, Bielefeld, March 2007:
      
   5. Variable Selection Algorithm for Micro Array Data:
      The analysis of gene expression data is currently a very challenging task. However this paper shows that we can use CMAT to find a very small number of genes from 22283 genes of an Affymetrix chip which yields an exact classification of two kinds of cancer.
   6. On the new CFA model improvement algorithm
   7. On the difference of p values for exact logistic regression computed by SAS PROC LOGISTIC, elrm in R, and CMAT.

   ---

   Things ToDo

   CMAT was first written for Unix and later for Windows.
   1. Versions for Mac, Linux, and Unix (should be easy since C code is portable, and lex and yacc are native in Unix)
   2. Dynamic binding of C and Fortran users code.
   3. Extending preprocessor commands.

   ---

   New Developments Since Release 3 in December 2002

   • Newsletter January 2003
     1. Faster matrix concatenation
     2. Reading and Writing of Matlab version 5 .mat Files
     3. ISOREG(): Isotone Regression (PAVA: Pool Adjacent Violators Algorithm, Optimal Scaling)
     4. VARSEL(): Single and multiple response variable selection:
        Forward, backward, and stepwise selection. All subset combinations or randomly generated samples.
   • Newsletter March 2003
     1. SIGN2(a,b) and SIGN4(a1,a2,a3,b) signum functions
     2. CANCOR(): Canonical Correlation Analysis
     3. MBURG(): Modified Burg algorithm for one- and twodimensional time series
   • Newsletter May 2003
     1. GLIM(): fixed some bugs and added some observationwise stat and ROC curve
     2. GLMIXD(): fixed lots of bugs, added type 1 and type 3 estimates, and added some observationwise stat and ROC curve
     3. CDF23(): 2 and 3 dimensional quadrature of normal and t distribution
     4. PROMEP(): experimental design
     5. NOHARM(): factor analysis for dichotomous (0,1) data with robust (nonnormal) GOF and ASEs
     6. FACTOR(): exploratory factor analysis with robust (nonnormal) GOF and ASEs
   • Newsletter July 2003
     1. SEM(): robust asympt. standard errors and robust Satorra-Bentler Chisquare
        Jackknife for identifying model outliers ("misfits")
     2. SVM(): automatic parameter tuning and two new methods (NSVM and FSM)
        block, split, and random cross validation
        Jackknife for identifying model outliers ("misfits")
   • Newsletter September 2003
     1. POLYCHOR(): polychoric correlation matrices and their covariance matrix
     2. ODE(): two new algorithms
     3. TRI2VEC(): move triangular (lower or upper) part of matrix to vector
     4. GENEREAD(): reading microarray data
   • Newsletter November 2003
     1. KSTEST(): new features for Kolmogorov-Smirnov test
     2. KSPROB(): compute prob of Kolmogorov CDF
     3. HOTELL(): compute classic and robust one-sample Hotellings Test with confidence intervals
     4. PLS(): partial least squares (PLS) and principal components regression (PCR)
        block, split, and random cross validation
     5. SVDTRIP(): compute singular triplets(U,D,V) for large and/or sparse matrices
   • Newsletter January 2004
     1. SIR(): sliced inverse regression for dimension reduction
        (including principal Hessian directions)
   • Newsletter June 2004
     1. PLS(): randomization test for number of components added
     2. SVM(): new feature selection methods added
     3. GAROTTE(): feature selection algorithm by Breiman (1993)
     4. LARS(): group of feature selection methods including the following:
        LARS: Least Angle Regression (Efron, Hastie, Johnstone & Tibshirani, 2002)
        Lasso: Tibshirani (1996), Osborne, Presnell & Turlach (2000),
        Foreward Stagewise: (Efron, Hastie, Johnstone & Tibshirani, 2002)
        Ridge Regression
        Elastic Net (Zou & Hastie; 2003)
        Univariate Soft Thresholding (Donoho et al., 1995)
     5. NLKPCA(): nonlinear kernel PCA (Schoelkopf, ; Rosipal & Trejo; 2001))
     6. NLKPLS(): nonlinear kernel PLS (Bennett & Embrechts; 2003)
   • Newsletter December 2004
     1. REG(): Jackknifing for outlier (misfit) detection added
     2. FROTATE(): many new rotation methods added (Bernaards & Jennrich, 2004)
     3. LRFORW(): linear LS forward selection method for very many variables
        (no need for storing the large X'X matrix)
     4. RANDISC(): some generators for discrete random variates (Marsaglia, Tsang, & Wang, 2004)
     5. SCREETST(): methods for testing the significant number of eigen values of X'X or covariance matrices or singular values of rectangular matrices
     6. SCALPHA(): computes the sample coefficient by Cronbach (1951) with asymptotic standard error and confidence interval
     7. SPLIT(): simple CART algorithm for binary response with Chisquare split criterion
     8. VARCLUS(): variable cluster algorithm for very many variables (similar to SAS PROC VARCLUS, but without the need to store the large X'X or covariance matrix)
   • Newsletter July 2005
     1. PCA(): implements eight different algorithms of principal component analysis including asymptotic standard errors and confidence intervals for unrotated and rotated component loadings (normal theory analytic and bootstrap)
     2. FACTOR(): asymptotic standard errors and confidence intervals for orthogonal and obliquely rotated factor solutions were added (normal theory analytic and bootstrap)
     3. NOHARM(): entire suite of rotation algorithms are added; asymptotic standard errors and confidence intervals for orthogonal and obliquely rotated factor solutions were added (normal theory analytic and bootstrap)
     4. CENTROID(): implements methods for classical centroid decomposition
     5. GENEREAD(): for comma (or otherwise) separated data set input
     6. HISTOGRM(): implements computation of histogram frequencies
     7. IMPUTE(): implements various methods for missing value imputation
     8. NNMF(): implements algorithm for nonnegative matrix factorization
     9. PERMUTE(): obtains permutations and combinations (all or stepwise)
     10. QUANTILE(): compute quantiles
     11. SIMDID(): compute some (not so great) similarity and distance measures
     12. SVDUPD(): rank-k update of the svd of a matrix
   • Newsletter December 2005
     1. GLIM(): multinomial Logit model is added
     2. ANACOR(): correspondence analysis of frequency tables (Gifi, 1990)
     3. ANAPROF(): correspondence analysis of profile data (Gifi, 1990)
     4. PRINCALS(): principal component analysis for categorical data (Gifi, 1990)
     5. CNDCOV(): conditional covariariance matrices (time series)
   • Newsletter July 2006
     1. CANALS(): canonical correlation analysis of two sets of variables (Gifi, 1990)
     2. CUCLCR(): cubic cluster criterion and R2 for given cluster decomposition (Sarle, 1983)
     3. DEMREG(): univariate Deming regression
     4. DIXONR(): pdf, cdf and critical values for Dixon's r
     5. FICA(): (Fast) Independent Component Analysis
     6. HISTPLOT(): plotting a histogram
     7. HOMALS(): homogeneity analysis of categorical data (Gifi, 1990)
     8. ITA(): classical and inductive item tree analysis (Schrepp, 2006)
     9. OVERALS(): canonical correlation analysis of more than two sets of variables (Gifi, 1990)
     10. PRIMALS(): one-dimensional homogeneity analysis of categorical data (Gifi, 1990)
     11. SDD(): SemiDiscrete Decomposition (Kolda & O'Leary, 1999)
     12. SGMANOVA(): multivariate analysis of variance based on spatial signs signs (robust estimation method)
     13. XYPLOT(): plotting a X-Y diagram
     14. ZOVERW(): probability and density of the normal distributed ratio z/w of normal distributed variables z and w (Marsaglia, 2006)
   • Newsletter December 2006
     1. BYTE(): transfers integers into characters using the ASCI table
     2. BRANKS(): computes tied and bivariate ranks
     3. COVLAG(): computes autocovariance estimates for a vector time series
     4. GEE(): generalized estimation of equations (Liang and Zeeger)
     5. RANKTIE(): averaging tie ranking of entries of a vector
     6. ROCCOMP(): test the equality of areas under ROC curve (DeLong et al, 1988)
     7. SCAD(): Smoothly Clipped Absolute Deviations (Fan \& Li, 2002) for LS regression, PH regression, and SVM
     8. SMSVM(): (structured) multicategory SVM (Lee, LIN, and Wahba, 2003)
     9. TOEPLITZ(): generates (block) Toeplitz matrix
   • Newsletter July 2007
     1. Language extension for multidimensional arrays and implementing tensor operations
     2. Language extension for lists of data objects
     3. CONST(): multidimensional extension of CONS()
     4. COVSHRK(): shrinking the covariance matrix for outliers in data
     5. DIM(): returning sizes of dimension of data objects (vectors, matrices, tensors)
     6. DIMLABEL(): assigning or pulling labels from tensor dimensions
     7. DIMNAME(): assigning or pulling names from tensor dimensions
     8. LOC(): returning index locations of specific data entries
     9. MAT2TEN(): create tensor from list of matrices
     10. RANDT(): multidimensional extension of RAND()
     11. TEN2MAT(): create list of matrices from tensor
     12. TEN2VEC(): move entries of tensor into data vector
     13. VEC2TEN(): move entries of data vector into tensor
     14. TENPERM(): reorder (permute) tensor dimensions
     15. TENTVEC(): multiply tensor with vector or list of vectors
     16. TENTMAT(): multiply tensor with matrix or list of matrices
     17. TENTTEN(): multiply tensor with tensor
     18. Extension to MAX() and MIN() functions
     19. Extension to SCAD function
     20. Extension to SMSVM() function (Lee, Lin, and Wahba, 2003)
     21. Extension to SVM() function
     22. Two types of index processing in matrices
     23. Fixing bugs for new release 4
   • Newsletter December 2007
     1. DIM(): returns dimensionality of data object
     2. IRTML(): maximum likelihood IRT (item response theory)
     3. IRTMS(): Mokken scale IRT
     4. MDS(): multidimensional scaling and unfolding
     5. SETDIFF(): set difference of two data objects
     6. SETISECT(): set intersection of two data objects
     7. SETMEMBR(): returns binary membership of the entries of one object in another one
     8. SETUNION(): set union of two data objects
     9. SETXOR(): set XOR (eXclusive OR) of two data objects
     10. SIZE(): returns size in dimensions of data object
     11. SORTROW(): sorts rows of a data matrix w.r.t to sorting key
     12. UNIQUE(): returns the the unique entries of a data object
     13. VEC2TRI(): moves vector into compact triangular matrix
   • Newsletter July 2008
     1. CODAPP(): applying the complete orthogonal decomposition
     2. HBADDTST(), HBANOVA(), HBBARTLETT(), HBCOVAR(), HBDISCRIM(), HBLRG(), HBLTST(), HBRCMP()
     3. SELC():
     4. URD1OUT():
     5. Extension of MIN() and MAX() functions for more than two arguments
     6. Extension of IRTML(): input probability data and R1 measure
     7. extensions for COD(), MDS, and ODE() functions
     8. Extension of NLP() and NLE(): more return arguments
     9. Extension of NLP(): grid search
     10. Extension of NLP(): new conjuage gradient techniques: Birgin-Martinez and scaled PR and FR
     11. Many test examples for NLP() in Part II
   • Newsletter December 2008
     1. Extension of NLP(): UOBYQA and NEWUOA algorithms
     2. Extension of NLP(): Nonsmooth BT algorithms
     3. LOG2(): logarithm w.r.t. base 2
     4. ENCRYPT() and DECRYPT(): for encrypting files and directories
     5. LOCATN(): algorithms for the optimal location assignment problem (greedy algorithm and Langrangean relaxation)
     6. NLFIT(): data mining using stagewise nonlinear regression using sets of activation and link functions
     7. NLFITPRD(): scoring a data set using the model from NLFIT()
     8. Some algorithms for normalizing microarray data
     9. Comparing the performance of random generators for normal distribution
     10. Application of the location assignment for matching the performance of an index fund
     11. Testing CMAT for release 5 in January 2009
   • Newsletter July 2009
     1. Fixed bugs in MRAND()
     2. Extension to the CMAT language: Adding k-dimensional trees to the set of data objects:
        1. KDTCRT(): create k-dimensional tree from data matrix
        2. KDTNEA(): obtain nearest neighbor nodes of kD tree
        3. KDTRNG(): obtain nodes of kD tree inside ball with specified radius
     3. Extensions to AFFVSN():
     4. Extensions to GLIM(): Hosmer and Lemeshow Test
     5. Extensions to CLUSTER(): more and better returns, some plotting
     6. Extensions to TOEPLITZ(): Levinson, Trench, and Durbin algorithm added for solving the Yule-Walker equations
     7. Extensions to UNIVAR(): many more location and scale measures
     8. AFFRMA(): Robust Multichip Average (Bolstad et.al, 2003) (not finished yet)
     9. ARIMA(): is still in the works
     10. ARMCOV(): modified covariance method for liner time series prediction
     11. BURG(): compute moving average whitening filter using method by Burg (1968)
     12. LDP(): linear distance programming (Lawson and Hanson, 1995)
     13. LOESS(): multivariate robust locally weighted regression (Cleveland, Grosse, and Shyu, 1992)
     14. LOWES(): univariate robust locally weighted regression (Cleveland, 1979)
     15. MEMPSD(): compute power spectrum of autoregressive filter
     16. POLYFIT(): fitting the polynomial model
     17. POLYVAL(): evaluating the polynomial model
     18. PPPD(): compute percentage points of Pearson distribution
     19. PWELCH(): compute power spectrum by periodogram method (Welch, 1967)
     20. SAMPLE(): equal and unequal probability sampling with or without replacement
     21. SORTP(): partial sorting for quantile
     22. STAND(): columnwise standardization of numeric matrix wrt. location and scale
     23. TSLOCFOR(): forecasting zero or first order local model
     24. TSLOCTST(): error testing of zero or first order local model
     25. TSMEAS(): for a large variety of time series measurements
     26. TSTRANS(): for a variety of time series data transformations, surrogates, and filters
     27. X11(): seasonal smoothing of monthly or quarterly time series data
     28. Some functions for combinatorics:
         1. COMBN(): generate all combinations of m elements taken n at the time
         2. DMNOM(): density of multinomial distribution
         3. HCUBE(): generate all points on hypercube lattice
         4. RMULT(): random generator for multinomial distribution (similar to MRAND())
         5. NSIMPLEX(): get number of points on (p,n) simplex
         6. XSIMPLEX(): generate all points on (p,n) simplex
     29. Illustrating Hosmer and Lemeshow Test
   • Newsletter December 2009
     1. Extensions to NLP(): new option for nonlinear constraints and new methods (e.g. Simulated Annealing, subgradient methods)
     2. Extensions to TSMEAS(): new functions added:
        • Partial autocorrelations with robust asymptotic standard errors
        • Ljung-Box test for serial correlation
        • (Robust) LM test for serial correlation
        • Newey-West covariance matrix
        • OLS Regression with Newey-West (HAC) as. standard errors
        • (Augmented) Dickey-Fuller testing
        • Granger causality testing (Likelihood-Ratio, LM, and Wald inference)
        • Vector AR modeling (homo- and heteroskedastic, correlated and uncorrelated errors)
        • Impulse response modeling for specified lead (homo- and heteroskedastic, correlated and uncorrelated erros)
     3. Extensions to TSTRANS(): new transformations added :
        • Box-Cox transform
        • Lag and Log transform
        • Baxter-King filtering
        • Hodrick-Prescott filtering
     4. Extensions to KSTEST(): many new distributions added, additional returns
     5. BERKOW(): Berkowitz testing for time series or cross sectional data for distributions like KSTEST()
     6. ARMA(): ML estimation of the AutoRegressive Moving Average model
     7. ARMAFORE(): Forecasting using ARMA model estimates
     8. ARHETERO(): Heterogeneous AR model estimation
     9. GARCH(): ARCH, GARCH, TARCH, AVARCH, ZARCH, APARCH, EGARCH, AGARCH, NAGARCH, IGARCH, FIGARCH model estimation
     10. JARBERA(): Jarque-Bera test for normal distribution
     11. MUCOMP(): comparing different hypotheses (model restrictions) for linearly constrained ("confirmatory") ANOVA (Kuiper, Klugkist, & Hojtink)
     12. SHAPWILK(): Shapiro-Wilk test for normal distribution
   • Newsletter July 2010
     1. Extending val = MAX(a,...) to < val,ind > = MAX(a,...) and the same for MIN()
     2. Extending NNMF() for symmetric nonnegative matrix factorization, C=HH' and C=HSH'
     3. Extending NNMF() for (left, right, and bi-) orthogonal nonnegative matrix factorization, C=UH' and C=USH'
     4. Extending NNMF() for orthogonal symmetric nonnegative matrix factorization, C=HH' and C=HSH'
     5. Creating a large number of links for the Reference Manual
     6. The gnuplot ... gpend syntax for interactive gnuplot input
     7. BORUTA(): Variable selection algorithm (wrapper of Random Forest; Kursa & Rudnicki)
     8. GPBATCH(): Running gnuplot input scripts in batch mode
     9. HELP(): Opening the Reference Manual at specific bookmark terms
     10. < val,ind > = MAXN(a,n) and < val,ind > = MAXN(a,n) for the n largest resp. smallest values of a
     11. RANFOR(): Random Forest algorithm for Classification and Regression (Breiman)
     12. RAFPRD(): Scoring the Random Forest model for Classification and Regression
     13. PROPURS(): Projected Pursuit PCA (Friedman & Tukey)
     14. SURVCURV(): Survival curves: Adjusted for Cox PH and Aalen's Model (Zhang et.al, 2007)
     15. SURVREG(): Survival regression: Cox proportional Hazards Model, Aalen's additive model, GLIM models (extreme, logistic, Gaussian; Weibull, loglog, lognormal, exponential, Rayleigh)
     16. SYSTEM(): Execute shell commands, save output in string data
     17. SPAWN(): Execute child process (with or without reentry or running concurrently)
     18. ZIP7(): (Encrypted) Compressing and decompressing using the 7-zip program
   • Newsletter December 2010
     1. Modifying LRFORW(): options matrix input argument
     2. Extending LOC(): for indices of missing values
     3. Extending NLREG(): permitting simple boundary, linear, and nonlinear constraints and specifying derivatives
     4. AUROC(): area under the ROC curve with asymptotic standard errors
     5. DELTA(): Delta method for computing asymptotic standard errors
     6. HBTTEST(): various forms of t test
     7. LRALLV(): all variables subsets regression (full enumeration and stochastic search)
     8. NOBLANKS(): removing leading and trailing blanks in string data
     9. ORDER(): hierarchical ranking in tied obsservations (similar to R function)
     10. SMP(): stochastic matching pursuit and componentwise Gibbs sampler for variable selection (Chen et.al)
     11. SURVCURV(): Survival curves: common types: Kaplan-Meier, Fleming-Harrington, Tsiatis, Aalen, Kalbfleisch-Prentice, Greenwood etc.
     12. SURVFOR(): Survival forest (not finished yet)
     13. SURVPRD(): Survival regression test set scoring (prediction and residuals)
   • Newsletter July 2011
     1. BIDIMREG(): Bidimensional Regression between 2-dimensional configurations (Tobler, 1994)
     2. CFA(): Confirmatory Factor Analysis (categorical data, robust, with automatic model search)
     3. DEG2RAD() and RAD2DEG(): conversion between degrees and radians
     4. INVUPD(): rank-1 update of the inverse of a pd matrix
     5. SOM(): Self-Organizing Maps, supervised Kohonen networks
   • Newsletter December 2011
     1. Now the input of hexadecimal numbers is permitted (transformed into unsigned long integers)
     2. Extended REPLACE(): for multiple replacements in scalars, vectors, matrices, or tensors
     3. Optional second argument for SRAND() for initialization of specific uniform generators
     4. Added: uniform random generators to RAND() function:
        • Mersenne-Twister(Matsumoto and Nishimura, 1998)
        • Advanced Encryption Standard (ASE) (Rijndael)
        • GFSR4 (Ziff, 1998)
        • RANLUX (Luescher, 1994)
     5. Worked over: MRAND() for multivariate random number generation
     6. New distributions for multivariate random number generation: t(mu,sigma,df), Pearson, Khintchine
     7. Worked over tests for univariate normality: Kolmogorov-Smirnov, Anderson-Darling, Shapiro-Wilks, Jarqe-Bera
     8. MVNTEST(): Various tests of multivariate normality: Mardia's tests of MV skewness and kurtosis, Royston, Henze-Zirkler, Doornik-Hansen, Small, Mudholkar, etc.
     9. CPERM(): permutation of columns of a matrix
     10. RPERM(): permutation of rows of a matrix
     11. ASSOC(): data mining items (Agrarwal, Imielinski, and Swami, 1993)
     12. RULES(): data mining items (Agrarwal, Imielinski, and Swami, 1993)
     13. SEQU(): data mining items (Agrarwal, Imielinski, and Swami, 1993)
     14. HANKEL(): create Hankel matrices
     15. CHNGTXT(): changes in string data (scalars, vectors, matrices, and tensors)
     16. CONVHULL(): convex hull of 2-, 3-, d-dimensional point configurations (Barber, Dobkin, and Huhdanpaa, 1996)
     17. DELAUNAY(): Delaunay triangulation (Barber, Dobkin, and Huhdanpaa, 1996)
     18. VORONIN(): Voronin diagrams (Barber, Dobkin, and Huhdanpaa, 1996)
   • Newsletter July 2012
     1. Worked over: ENCRYPT() and DECRYPT(): more methods (AES, SHA2,...) and vector input for names of files and directories
     2. CSVREAD(): Reading of CSV (Comma-Separated-Values) files
     3. ENCRYP2(), DECRYP2(): similar to ENCRYPT() and DECRYPT(), however treats string objects (not files or directories)
     4. RECUPAR(): Recursive partitioning (tree splitting similar to SAS Macro TREEDISC, but much faster)
     5. INSERT(): with arguments compatible with SAS/IML function
     6. MIXREGV(): implementation of Don Hedeker's Fortran program MIXREGLS
     7. REMOVE(): with arguments compatible with SAS/IML function
     8. Almost all string functions (str...) extended for vector, matrix, and tensor arguments
   • Newsletter December 2012 and July 2013
     1. Changed to MS Visual C/C++ 2010 and Intel Parallel Studio XE 2013
     2. Worked over polychoric correlations for multiple sample applications, and functions HBTTEST(), SORTROW(),
     3. ENCRYPT(): added SHA-3 method to
     4. HORNER(): efficient evaluation of polynomials
     5. KDE() : 1- and 2-dimensional kernel density estimation
     6. MVSVM(): stepwise multivariate SVM (Thayanathan, 2005)
     7. PERMCOMB(): permutation and combination (stepwise)
     8. PREFNAME(): generate sets of prefix names
     9. RVM(): Relevance vector machine
     10. SDCSPM(): testing Srivstava's condition
     11. Application: Two-Phase Logistic Modeling
   • Newsletter December 2013
     1. Worked over TTEST()
     2. Worked over linear regression: REG(), LRFORW(), and LRALL()
     3. Worked over general linear modeling: GLMOD(), e.g. added multiple comparison techniques (Interfacing MULTCOMP())
     4. Worked over multivariate normal and t probabilities: CDFMVN()
     5. ICDFMV(): inverse CDF for multivariate normal and t distribution (given prob find quantile)
     6. MAHALANOBIS(): Mahalanobis distances (diagonal and full)
     7. MULTCOMP(): various parametric and nonparametric methods for multiple comparison of means and medians of K>2 samples
     8. PADJUST(): adjusted multivariate probabilities (e.g. Bonferroni, Holm, Hochberg)
     9. PDFMV(): PDF for multivariate normal and t distribution
     10. WILCOX(): Wilcoxon rank sum test and signed rank test comparing two samples
   • Newsletter July 2014
     1. Worked over LP(): interface with LPSOLVE, now with integer constraints and sensitivity analysis
     2. Added to NLP(): LINCOA (LINear Constraint Optimization Algorithm) to NLP() (M.J.D. Powell)
     3. BOUNDBOX(): compute smallest rectangular box surrounding a specified set of points
     4. HAMILTON(): find all or some Hamiltonian circuits in directional graphs
     5. KNAPSACK(): solving (approximately) the one- and multidimensional Knapsack problem
     6. LATLONG(): various computations with Latitude and Longitude data
     7. LOCAT1(): solves multifacility location problem
     8. LPASSIGN(): solving the linear assignment problem with LPSOLVE, LAPJV, or the Hungarian method and solving the linear bottleneck assignment problem with BOTJV
     9. LPTRANSP(): solving the linear transport problem with LPSOLVE
     10. MAXEMPTY(): finds coordinates and volume of largest empty box parallel to (x,y) axes and surrounded by a specified set of points
     11. MSTGRA(): Minimum Spanning Tree based on graph data (MSTREE was renamed into MSTDIS for Minimum Spanning Tree based on distance data)
     12. SPLNET(): compute shortest path (length) between two points of network (graph) data
     13. TSP(): various methods for solving the symmetric and asymmetric Traveling Salesman Problem (Interface to Linkern and Concorde)
   • Newsletter December 2014
     1. FILESTAT(): returns vector of file statistics
     2. MPSFILE(): transforming MPS file to matrix notation of LP specification and vice versa
     3. PRITFILE(): transfer text file rowwise into vector of strings
     4. CONTSIM(): Random Generation of contingency tables with specified row and column sums
     5. All LP algorithms worked over
     6. Interface to Clp (Coin Linear Programming) and Cbc (Coin Branch-and-Cut) in COIN-OR
     Some "Exact Statistics" for small samples:
     1. XCTBINOM(): exact binomial test: p values and confidence intervals
     2. XCTBIP1() : P(alt) for exact binomial test
     3. XCTBIPOW(): Power for exact binomial test
     4. XCTBISSZ(): Sample size for exact binomial test
     5. XCTPOISS(): exact Poisson test: p values and confidence intervals
     6. XCTFISHR(): Fisher's exact test for 2 x 2 contingency tables
     7. XCTFIPOW(): Power for Fisher's exact test for 2 x 2 contingency tables
     8. XCTFISSZ(): Sample size for Fisher's exact test for 2 x 2 contingency tables
     9. XCTLOG(): MCMC Method for exact logistic regression (see elrm)
     10. XCTMCNEM(): McNemar's exact test for 2 x 2 contingency tables
     11. XCTSIMU(): MC method of Fisher's exact test for m x n contingency tables
     12. XCTHYBR(): Hybrid method for Fisher's exact test of m x n contingency tables
   • Newsletter December 2015
     1. Extending data types to Structs, entries are referred by compound name separated by dot: struct_name.entry_name
     2. RANK() function extended for very large and sparse matrices
     3. Work on preprocessor for LP() function
     4. CLP interface for SVM() regression (Bi et.al, 2002) and SMSVM (Lee, Lin, and Wahba, 2003)
     5. RCCOUNT(): count the occurence of numeric or string values in rows and columns of a matrix
     6. SOUND(): playing sound (of specified frequency and duration) on speakers
   • Newsletter July 2016
     1. This is now the very carefully tested version 9 ready for downlowd moved to the internet at the end of September 2016. Maybe it is the most error free since version 4.
     2. Work on printed output of tensors and lists
     3. Work on function SVM() for parameter tuning
     4. FNAMPID(): concatenate filename and actual process ID
     5. FREMOVE(): remove file with specified name
     6. FRENAME(): rename file
     7. ERROR(): print error message into log
     8. LSTLABEL(): assigning labels to entries of data lists
     9. LSTNAME(): assigning names to entries of data lists
     10. OUTLIER(): various methods for finding outliers in univariate data
     11. PID(): returns integer process ID
     12. SVMFSM(): SVM one-step feature selection (sparse L1 model fit)
     13. SVMSTW(): SVM stepwise feature selection (forward and backward, for only linear kernel)
     14. WARNING(): print warning message into log
     15. Rename objects with RENAME statement

     ---

     Now, CMAT got some attention at Dilbert

     "Algorithm" by Paul Boggs

     Algorithm, my delight Running at the speed of light What the genius, who the geek Could forge thy objects ever sleek? In what language were thee writ That enabled every bit? What the templates - how applied And what the math down deep inside?

     Thou wert born with MPI And that enabled thee to fly On Beowulf we set thee free All cycles thou consumed with glee. When the stars come out at night And ask a sacrificial rite Do we just sneer and charge ahead With no fear and with no dread?

     All this wonder; all this speed And yet I wait here still in need Alas the run's untimely halt Said naught but that some seg did fault. Algorithm, my delight Running at the speed of light What the genius, who the geek Could forge thy objects ever sleek?

     ---

     Contact information:

     • email: cmat.wolfgang@gmail.com

     ---

     Back to Homepage