One of the most critical problems in Internet security is the Denial-of-Service (DoS) attack, which aims to make a service unavailable to legitimate clients. In this project, we consider a sophisticated attack, called service-level DoS attack, which is very difficult to identify as malicious requests can be made arbitrarily similar to legitimate ones and can bypass the network-based defense systems. We propose a novel framework to detect the attackers based on the group testing (GT) technique which can overcome the limitations of current detection approaches. More specifically, this project seeks to investigate the following challenges: 1) Dynamic threshold model is studied to handle the legitimate bursts and variance in the number of clients on each server; 2) Legitimates and malicious requests are similar, required new testing design without examining each request one by one or tightly specifying legitimate behaviors; 3) In addition, the study of the proposed model evokes a new type of GT, called Size Constraint Group Testing (SCGT) which requires an in depth analysis of matrix construction complexity. This mathematically rigorous framework helps to minimize the false positive and false negative of detection, which is the main problem currently for any existing defense mechanisms.