Correspondence

Immune Signatures in Follicular Lymphoma

N Engl J Med 2005; 352:1496-1497April 7, 2005

Article

To the Editor:

The recent article by Dave et al. (Nov. 18 issue)1 is an important study correlating gene expression with survival in follicular lymphoma. My colleagues and I have reexamined their findings.

Analysis with the use of standard tools such as Significance Analysis of Microarrays (SAM) suggests that there is little or no predictive power in these data. With help from the authors and after considerable effort, we reconstructed their analysis. Dave et al. divided their data at random into equal-sized training and test sets, applied a detailed model-fitting recipe to the training set, and finally evaluated the model's performance in predicting survival in the test set.

Their recipe gives a significant P value when used on the test set, but the result is extremely fragile. When the training and test sets are swapped, and the recipe is applied, their model does not emerge. In fact, no model is found that is significantly associated with survival when applied to the test set. As another example of fragility, if one makes small changes to some of the reasonable but arbitrary choices in their recipe, nothing significant emerges. Thus, our reanalysis sheds serious doubt on the reproducibility of the authors' biologic findings. (Full details may be found at www-stat.stanford.edu/~tibs/FL/report.)

Robert Tibshirani, Ph.D.
Stanford University, Stanford, CA 94305
tibs@stanford.edu

1 References

1. 1

   Dave SS, Wright G, Tan B, et al. Prediction of survival in follicular lymphoma based on molecular features of tumor-infiltrating immune cells. N Engl J Med 2004;351:2159-2169
   Full Text | Web of Science | Medline

To the Editor:

Dave et al. suggest that survival in follicular lymphoma correlates with two gene signatures, which they termed immune-response 1 and immune-response 2. Because most of the genes in these signatures were expressed more highly in CD19– cells, the authors conclude incorrectly that survival is based on features of the nonmalignant cells in the tumors. It must be noted that many of the genes were expressed at only slightly higher levels in CD19– cells. We estimate that the CD19– cells contributed more than 50 percent of the messenger RNA for 33 of the 43 immune-response 1 genes, but only 2 of the 24 immune-response 2 genes. When each signature was tested separately, survival in the test set correlated only with the immune-response 2 signature (Tibshirani R: personal communication). Thus, it is unwarranted to conclude that survival correlates with features in the nonmalignant cells, as claimed in the title of the paper.

Wan-Jen Hong, B.S.
Roger Warnke, M.D.
Gilbert Chu, M.D., Ph.D.
Stanford University School of Medicine, Stanford, CA 94305
chu@stanford.edu

Author/Editor Response

Nothing in Tibshirani's analysis calls into dispute the fact that we discovered and validated a strong association between gene expression in follicular lymphoma and overall survival. Our validation method is the accepted standard for supervised analyses of microarray data to create a survival predictor.1 Our predictor was highly associated with survival in the independent test set (P<0.001). Furthermore, we randomly selected half of our cases 2000 times and computed the association of our model with survival in each half-set (using model coefficients derived from the original training set). The median P value for these samplings was less than 0.001, and no half-set sampling of our data had a P value greater than 0.011 (Figure 1Figure 1Results of Samplings of Half-Sets and Association between Gene Expression and Survival.).

Our predictor could not have been discovered with the SAM method, which relies solely on univariate associations with survival. Rather, our predictor derives its strength from the synergistic combination of two gene-expression signatures in a multivariate model. Tibshirani confuses the ability of our method to discover a survival association with the fact that we actually found one that validated the association. When he exchanged the training set for the test set, he was unable to rediscover our gene-expression predictor because some genes in our predictor fell below the P value threshold for association with survival in the test set. This does not negate the fact that our model is highly associated with survival in the test set.

Hong et al. have made three errors. First, in our sorted subpopulations, the CD19– fraction contained, on average, 12.6 percent contamination with follicular-lymphoma cells, not 25 percent, as they claim. To believe that the higher expression of the immune-response signatures in the CD19– fraction is due to this 12.6 percent contamination requires that the lymphoma cells in the CD19– fraction have expression of the immune-response signatures that was more than eight times as high as that in their counterparts in the CD19+ fraction. Second, Hong et al. incorrectly discount the immune-response 1 signature, which contributes significantly to the survival model in the test set (P<0.001). Third, many of the immune-response signature genes are selectively expressed in T cells, monocytes, or dendritic cells, or in more than one of these, but not in B cells, making the contention of Hong et al. even more implausible.

Louis M. Staudt, M.D., Ph.D.
George Wright, Ph.D.
Sandeep Dave, M.D.
National Cancer Institute, Bethesda, MD 20892
lstaudt@mail.nih.gov

1 References

1. 1

   Ransohoff DF. Rules of evidence for cancer molecular-marker discovery and validation. Nat Rev Cancer 2004;4:309-314
   CrossRef | Web of Science | Medline

Citing Articles (11)

Citing Articles

1. 1

   Weijie Chen, Brandon D. Gallas, Waleed A. Yousef. (2012) Classifier variability: Accounting for training and testing. Pattern Recognition
   CrossRef

2. 2

   Elizabeth Purdom, Sach Mukherjee. 2011. Transcriptomic Technologies and Statistical Data Analysis. , 133-162.
   CrossRef

3. 3

   Weijie Chen, Waleed A. Yousef, Brandon D. Gallas, Elizabeth R. Hsu, Samir Lababidi, Rong Tang, Gene A. Pennello, W. Fraser Symmans, Lajos Pusztai. (2011) Uncertainty estimation with a finite dataset in the assessment of classification models. Computational Statistics & Data Analysis
   CrossRef

4. 4

   Andrea Janikova, Boris Tichy, Jana Supikova, Katerina Stano-Kozubik, Sarka Pospisilova, Leos Kren, Ingrid Vasova, David Salek, Jiri Mayer. (2011) Gene expression profiling in follicular lymphoma and its implication for clinical practice. Leukemia & Lymphoma 52:1, 59-68
   CrossRef

5. 5

   Leming Shi, Gregory Campbell, Wendell D Jones, Fabien Campagne, Zhining Wen, Stephen J Walker, Zhenqiang Su, Tzu-Ming Chu, Federico M Goodsaid, Lajos Pusztai, John D Shaughnessy, André Oberthuer, Russell S Thomas, Richard S Paules, Mark Fielden, Bart Barlogie, Weijie Chen, Pan Du, Matthias Fischer, Cesare Furlanello, Brandon D Gallas, Xijin Ge, Dalila B Megherbi, W Fraser Symmans, May D Wang, John Zhang, Hans Bitter, Benedikt Brors, Pierre R Bushel, Max Bylesjo, Minjun Chen, Jie Cheng, Jing Cheng, Jeff Chou, Timothy S Davison, Mauro Delorenzi, Youping Deng, Viswanath Devanarayan, David J Dix, Joaquin Dopazo, Kevin C Dorff, Fathi Elloumi, Jianqing Fan, Shicai Fan, Xiaohui Fan, Hong Fang, Nina Gonzaludo, Kenneth R Hess, Huixiao Hong, Jun Huan, Rafael A Irizarry, Richard Judson, Dilafruz Juraeva, Samir Lababidi, Christophe G Lambert, Li Li, Yanen Li, Zhen Li, Simon M Lin, Guozhen Liu, Edward K Lobenhofer, Jun Luo, Wen Luo, Matthew N McCall, Yuri Nikolsky, Gene A Pennello, Roger G Perkins, Reena Philip, Vlad Popovici, Nathan D Price, Feng Qian, Andreas Scherer, Tieliu Shi, Weiwei Shi, Jaeyun Sung, Danielle Thierry-Mieg, Jean Thierry-Mieg, Venkata Thodima, Johan Trygg, Lakshmi Vishnuvajjala, Sue Jane Wang, Jianping Wu, Yichao Wu, Qian Xie, Waleed A Yousef, Liang Zhang, Xuegong Zhang, Sheng Zhong, Yiming Zhou, Sheng Zhu, Dhivya Arasappan, Wenjun Bao, Anne Bergstrom Lucas, Frank Berthold, Richard J Brennan, Andreas Buness, Jennifer G Catalano, Chang Chang, Rong Chen, Yiyu Cheng, Jian Cui, Wendy Czika, Francesca Demichelis, Xutao Deng, Damir Dosymbekov, Roland Eils, Yang Feng, Jennifer Fostel, Stephanie Fulmer-Smentek, James C Fuscoe, Laurent Gatto, Weigong Ge, Darlene R Goldstein, Li Guo, Donald N Halbert, Jing Han, Stephen C Harris, Christos Hatzis, Damir Herman, Jianping Huang, Roderick V Jensen, Rui Jiang, Charles D Johnson, Giuseppe Jurman, Yvonne Kahlert, Sadik A Khuder, Matthias Kohl, Jianying Li, Li Li, Menglong Li, Quan-Zhen Li, Shao Li, Zhiguang Li, Jie Liu, Ying Liu, Zhichao Liu, Lu Meng, Manuel Madera, Francisco Martinez-Murillo, Ignacio Medina, Joseph Meehan, Kelci Miclaus, Richard A Moffitt, David Montaner, Piali Mukherjee, George J Mulligan, Padraic Neville, Tatiana Nikolskaya, Baitang Ning, Grier P Page, Joel Parker, R Mitchell Parry, Xuejun Peng, Ron L Peterson, John H Phan, Brian Quanz, Yi Ren, Samantha Riccadonna, Alan H Roter, Frank W Samuelson, Martin M Schumacher, Joseph D Shambaugh, Qiang Shi, Richard Shippy, Shengzhu Si, Aaron Smalter, Christos Sotiriou, Mat Soukup, Frank Staedtler, Guido Steiner, Todd H Stokes, Qinglan Sun, Pei-Yi Tan, Rong Tang, Zivana Tezak, Brett Thorn, Marina Tsyganova, Yaron Turpaz, Silvia C Vega, Roberto Visintainer, Juergen von Frese, Charles Wang, Eric Wang, Junwei Wang, Wei Wang, Frank Westermann, James C Willey, Matthew Woods, Shujian Wu, Nianqing Xiao, Joshua Xu, Lei Xu, Lun Yang, Xiao Zeng, Jialu Zhang, Li Zhang, Min Zhang, Chen Zhao, Raj K Puri, Uwe Scherf, Weida Tong, Russell D Wolfinger. (2010) The MicroArray Quality Control (MAQC)-II study of common practices for the development and validation of microarray-based predictive models. Nature Biotechnology 28:8, 827-838
   CrossRef

6. 6

   G. Alex Bishop, Rohan B. H. Williams, Kris Kowdley, Geoffrey McCaughan, Christian Trautwein. (2008) Screening liver transplant patients for tolerance. Hepatology 48:6, 2082-2084
   CrossRef

7. 7

   B GALLAS, D BROWN. (2008) Reader studies for validation of CAD systems. Neural Networks 21:2-3, 387-397
   CrossRef

8. 8

   David F. Ransohoff. (2007) How to improve reliability and efficiency of research about molecular markers: roles of phases, guidelines, and study design. Journal of Clinical Epidemiology 60:12, 1205-1219
   CrossRef

9. 9

   Robert F. Wagner. (2007) From medical images to multiple-biomarker microarrays. Medical Physics 34:12, 4944
   CrossRef

10. 10

    Hanlee Ji, Ronald W Davis. (2006) Data quality in genomics and microarrays. Nature Biotechnology 24:9, 1112-1113
    CrossRef

11. 11

    J.H.G.M. van Beek. (2006) Channeling the Data Flood: Handling Large-Scale Biomolecular Measurements in Silico. Proceedings of the IEEE 94:4, 692-709
    CrossRef