Baidu has been caught cheating in the recent ImageNet competition（百度在ImageNet竞赛中作弊被抓现行）

Chinese search engine giantBaidu recently made headlines when its supercomputer reportedly beat out challengers from both Google andMicrosoft on the ImageNet image recognition test . However, the company has had to back down from those claims and issue an apology after details emerged suggesting that its success resulted from a scheme to cheat the testing system. As such, Baidu's accomplishment has been stricken from the books and the company has been banned from ImageNet challenges for a full year.

The issue began in Mid-May when Baidu claimed to have scored a record low 4.58% error rate on the test . This exam looks at how well computing clusters can identify objects and locations within photographs -- basically the technology behind Google Photo's auto-tagging feature -- except on large-scale file sets. Microsoft and Google, on the other hand scored 4.94 and 4.8 percent error rates, respectively. That's actually a bit better than the 5 percent average trained humans can achieve and a huge deal for the industry.

However on Tuesday, researchers who actually administered the ImageNet test called shenanigans on Baidu for setting up a series of dummy accounts to brute force a successful test run. The test rules state specifically that contestants are allowed to submit only two sets of test results each week. Baidu apparently set up 30 accounts and spammed the service with 200 requests in six months, 40 of which came over a single five-day period in March. Doing so potentially allowed Baidu engineers to artificially increase the recognition rate by "tuning" their software to the existing test data sets.

"This is pretty bad, and it is exactly why there is a held-out test set for the competition that is hosted on a separate server with limited access," Matthew Zeiler, CEO of AI software company Clarifai, told the Wall Street Journal . "If you know the test set, then you can tweak your parameters of your model however you want to optimize the test set."

In response, Baidu has issued a formal apology for its actions. If, you think apology is a good description for calling the incident a "mistake" and refusing to provide any additional details or explanation as to why it happened.

Statement from Baidu:

Dear ILSVRC community,

Recently the ILSVRC organizers contacted the Heterogeneous Computing team to inform us that we exceeded the allowable number of weekly submissions to the ImageNet servers (~ 200 submissions during the lifespan of our project).

We apologize for this mistake and are continuing to review the results. We have added a note to our research paper, Deep Image: Scaling up Image Recognition, and will continue to provide relevant updates as we learn more.

We are staunch supporters of fairness and transparency in the ImageNet Challenge and are committed to the integrity of the scientific process.

Ren Wu – Baidu Heterogeneous Computing Team

On May 17, 2015, the organizers of the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) contacted Baidu’s Heterogeneous Computing team, a five-member computer systems engineering group, to inform them that they had exceeded the allowable weekly number of submissions to the ImageNet server.

Immediately, we launched our own inquiry to understand what had happened. While the inquiry was underway, we specifically refrained from making public comments to the press and on social media, out of fairness to both the Baidu employees and the ImageNet organizers.  This week, we concluded the inquiry. We found that a team leader had directed junior engineers to submit more than two submissions per week, a breach of the current ImageNet rules.

Any action that runs counter to the highest standards of academic and scientific integrity, no matter how large or small, is unacceptable to us and does not reflect the culture of our company. We have zero tolerance for such behavior and have terminated the team leader’s employment.

During the inquiry, we verified that the Heterogeneous Computing team’s work was not connected in any way to image recognition research performed by other teams at Baidu. We have put in place new company-wide policies to ensure that teams and individuals understand and adhere to the highest standards. The new policies include employee training sessions and the establishment of a scientific advisory panel that will serve as a resource to our worldwide engineering teams moving forward.

Distinguished Scientist
Baidu, Inc.
August 2013 – Present (1 year 11 months)Institute of Deep Learning (IDL)
Big data + Deep learning + High performance computing => Intelligence
Big data + Deep learning + Heterogeneous computing => Success

Two of my recent keynotes explained above formulas well,

EnterpriseHPC 2014
HPC – Turning Big Data Into Deep Insight
http://www.datanami.com/2014/09/22/baidu-uses-deep-learning-drive-success-web/

Embedded Vision
Harnessing Heterogeneous Computing for Cloud- and Mobile-based Visual Search
https://www.youtube.com/watch?v=AX1TJbTlhgs

An early interview on CUDA Spotlight (May 16th, 2014)
http://www.nvidia.com/content/cuda/spotlights/ren-wu-baidu.html

My talk at GTC14 - "Deep Learning Meets Heterogeneous Computing"
http://on-demand.gputechconf.com/gtc/2014/presentations/S4651-deep-learning-meets-heterogeneous-computing.pdf
Advancements in Deep Learning and AI: Stepping inside Baidu’s “Brain”
Advancements in Deep Learning and AI: Stepping inside Baidu’s “Brain”

View On SlideShare
GTC15 Talk - DeepImage: Scaling up Image Recognition
GTC15 Talk - DeepImage: Scaling up Image Recognition

View On SlideShare
Lead Architect for HSA Server Software, Principal Member of Technical Staff
AMD
May 2012 – August 2013 (1 year 4 months)
Lead Architect for HSA Server Software
Principal Member of Technical Staff

Drive AMD's HSA server software, with special focus on cloud computing, big data analytics and high-performance computing.

For background information about AMD's HSA, see this HPCWire article
http://www.hpcwire.com/hpcwire/2012-05-07/amd_:_the_integration_revolution_.html

Want to know more? Join me at AMD’s Fusion12 Developer Summit, amd.com/afds
Principal Investigator of the CUDA Research Center - GPU Accelerated Scientist at HP
Hewlett-Packard Laboratories
August 2010 – May 2012 (1 year 10 months)
Invited to give a talk at GTC Asia in Beijing, Dec 14-15, 2011,

GPUs Push Envelope on BI Performance
http://www.datanami.com/datanami/2011-12-21/gpus_push_envelope_on_bi_performance.html?featured=top

Recent work on text analytics ( very large scale, high dimensional clustering )

http://www.hpcwire.com/hpcwire/2011-05-23/pittsburgh_supercomputing_center_accelerates_machine_learning_with_gpus.html
http://www.hpcwire.com/hpcwire/2011-05-26/the_weekly_top_five.html
http://www.hpcwire.com/hpcwire/2011-05-26/a_tale_of_two_gpu_computing_models.html


A recent interview by NVidia

CUDA Spotlight: GPU-Accelerated Large-Scale Analytics
http://developer.nvidia.com/cuda-spotlight-gpu-accelerated-large-scale-analytics


Few selected projects:

GPU-Acceleration of Seismic Imaging Algorithms
Large scale sensor analytics (~2k nodes for 5 weeks), real world application, tremendous speed-up (> 30x).
Seismic imaging, parallel and distributed algorithms, Very large scale convolution, FFT, and GPU acceleration

GPU-Accelerated Large Scale Analytics
Very high performance GPU algorithms for very large scale analytics.
Clustering algorithms, K-means, predicate evaluation, column scan, micro-sort, and regression clustering.
Senior Research Scientist
Hewlett Packard
August 2000 – August 2010 (10 years 1 month)
Few selected projects:

Analytics Inside
To push the computation more closer to the data
DBMS, UDF, massively parallelized analytics, and GPU acceleration

Illuminant Detection
Lead the effort to improve the illuminant detection algorithms.
Heuristic algorithm, machine learning, and neural networks

Currency Hedging
To help HP Treasury figuring out the optimal hedging strategies in Europe and Asia.
Large scale smart agent simulation, experimental economics, game theory and computational intelligence.

HP Revenue Modelling and Prediction
Data-driven machine learning system for quarterly revenue forecasting and prediction. System was online since Q303 for HP executives.
Machine learning, neural networks, heuristic algorithms.

HP Experimental Economics Software Platform
Project management, software engineering

HP San Designer
Heuristic algorithms for storage networks design
Performance tuning, heuristic algorithm, SAN design.
Team Lead - Data-intensive high-performance computing
HP Labs China
July 2006 – May 2008 (1 year 11 months)
1. Help to form HP Labs China
2. To build Data-intensive high-performance computing team that support web-analytics
R & D
Teradyne
May 1998 – August 2000 (2 years 4 months)