Authors Dokuparthi Prasanthi V. Rama Krishna
License CC-BY
International Journal of Science and Research (IJSR)
ISSN (Online): 2319-7064
Impact Factor (2012): 3.358
Malware Detection and Tracer Approach for
Operating System
Dokuparthi Prasanthi1, V. Rama Krishna2
1
M.Tech student, Department of CSE, Anurag Group of Institutions, Hyderabad, India
2
Assistant Professor, Department of CSE, Anurag Group of Institutions, Hyderabad, India
Abstract: Modern computer systems are built on a foundation of software components from a variety of vendors. While critical
applications might undergo extensive testing and evaluation procedures, the heterogeneity of software sources hazardous the integrity of
the execution environment for these trusted programs. For instance, if an attacker can be able to merge an application exploit with
privilege increase vulnerability, the Operating System (OS) can become corrupted. Mandatory Access Control (MAC) in a commercial
operating system to handle malware problem is a challenge but also a capable approach. The firmest barriers to apply MAC to defeat
malware programs are the incompatible and unusable problems in existing MAC systems. The aim of our study is to address these issues
design a novel Efficient Malware Detection and Tracer design (EMDT) using Hidden Markov model, which incorporates intrusion
detection and tracing in an operating system. In this proposed approach conceptually consists of three actions: tracing, detecting and
restricting deduced intruders .The novelty of the proposed study is that it leverages light-weight intrusion detection and tracing
techniques to automate security label configuration that is widely acknowledged as a tough issue when applying a MAC system in
practice. The other is that, rather than controlling information flow as a traditional MAC does, it traces intruders and restricts only their
significant malware behaviours’, where intruders characterize processes and executables that are potential agents of a remote attacker.
Our prototyping and testing’s on Windows operating system show that Tracer can effectively defeat all malware samples tested via
blocking malware behaviours while not causing a significant compatibility problem.
Keywords: Detection, intrusion, malware, tracing, vulnerability.
1. Introduction Each process or executable has two states, suspicious or
benign. The contributions of this study are We introduce
Malicious software (i.e., Malware) is one of the most severe EMDT, a novel MAC enforcement approach which
computer security problems today. A network of hosts which combines intrusion detection and tracing techniques to
are cooperated by malware and controlled by attackers can disable malware on a commercial OS in a compatible and
cause a lot of damages to information systems. As a useful usable manner. We have implemented EMDT to immobilize
malware defence technology, Mandatory Access Control malware timely without need of malware signatures. We
(MAC) works without relying on malware signatures and investigate the root reason so discover compatibility and low
blocks malware behaviours before they cause security usability problems of existing MACs. Although not all the
damage. Even if an unauthorized user manages to breach observations are brand new, we consider that understanding
other layers of defence, MAC is capable of act as the last these reasons more comprehensively and illustrating them
shelter to avoid the entire host from being compromised. through the design of an actual system are useful for other
However, MAC mechanisms built in commercial operating MAC researchers.
systems (OS) often go through from two problems which
make general users unenthusiastic to assume them. One 2. Related Work
problem is that a built-in MAC is mismatched with a lot of
application software and thus interferes with their running DTE proposed by Badger et al. (1995) is a classical MAC
and the other problem is low usability, which creates it model to confine process execution, which group’s processes
difficult to configure MAC properly. Our observations are as and files into domains and types, respectively and controls
follows: The incompatibility problem is introduced because accesses among domains and types. Tracer can be
the security labels of existing MACs are not capable to considered as a simplified DTE that has two domains (i.e.,
distinguish between malicious and benign entities, which benign and suspicious) and four types (i.e., benign, read-
Causes a enormous number of False Positives (FP) (i.e., protected, write-protected and suspicious). Moreover, Tracer
treating benign operations as malicious) thus avoiding many can usually configure the DTE attributes (i.e., domain and
benign software from performing legal operations; the low- type) of processes and files under the maintain of intrusion
usability problem is launched, because existing MACs are detection and tracing so as to develop usability. PRECIP
not capable to automatically label the huge number of Wang et al. (2008) addresses several practical issues that are
entities in OS and thus require tough configuration work at significant to contain spyware that be determined to leak
End users. With these investigation results, our main sensitive information. The risk-adaptive access controls
objective is to propose a novel MAC enforcement approach (Kaspersky Lab, 2012). That targets to create access control
EMDT, this consists of three actions: Detection, tracing and more dynamic so as to attain a better tradeoff between risk
restriction. and benefit. Most existing antimalware technologies are
based on detection (Kirda et al., 2006; Martignoni et al.,
2008). Tracer tries to combine detection and access control
Volume 3 Issue 9, September 2014
www.ijsr.net
Paper ID: SEP14492 1779
Licensed Under Creative Commons Attribution CC BY
International Journal of Science and Research (IJSR)
ISSN (Online): 2319-7064
Impact Factor (2012): 3.358
so that it not only can detect but also can block malware usage conventions unchanged.
behaviours before their harming security. Antimalware
technology that be similar to Proposed EMDT is behaviour
blocking (Nachenberg, 2002) which can confine the
behaviours of definite adverse programs that are profiled in
advance. Many commercial antimalware tools (Kaspersky
Lab, 2012; Viper Inc., 2012) also have a behaviour-based
module to protect against unknown malware programs.
Problems in MAC
Incompatibility is a familiar problem when enforcing a MAC
modeling operating system (Li et al., 2007; Fraser, 2000;
Wang et al., 2008). To examine its core reason, in a secure
network environment, we set up two mechanisms to run
MAC enforced operating systems with MLS policy allowed
and MAC module allowed. After a few days, we surveyed
that these MAC systems produced a enormous number of log Figure 1: EMDT Overview
records about denied accesses, which specified that some
applications failed and some acted irregularly. As the The above Fig. 1 gives an overview of EMDT which
operation environment is secure without intrusion and consists of three types of actions, tracing, detection and
malware, these denied accesses are thus “false positive.” restriction. Each process or executable has two states,
However, from the view of intrusion thwarting, these suspicious and benign. The restriction action forbids a
processes do not necessarily represent intruders so that their suspected intruder to make malware behaviours in order to
“read” or “write” accesses to the/tmp should not be merely guard CIAP. That is to protect integrity, confidentiality and
denied. Although it is possible to resolve this problem by availability, as well as to stop malware propagation. The
adding “hiding sub directories” under/tmp, it is still difficult three actions study as follows: Once detecting a suspected
to eliminate the FPs resulting from many other shared process or executable, EMDT labels it as suspicious and
entities on an OS Relying on these labels, a MAC system traces its descendent and interacted processes, as well as its
habitually fails to make correct decisions on intrusion generated executables. EMDT does not restrict benign
blocking which eventually results in many FPs. Low processes at all and permits suspicious processes to run as
usability is another problem in a MAC-enabled system, as it long as possible but stops their malware behaviours that
often requires make difficult configurations and would cause security damages.
unconventional ways of usage.
3. Proposed System
Efficient Malware Detection and Tracer (EMDT): In this
section, we present our EMDT approach that aims to
immobilize malware in a OS by disagreeing malware
behaviours. The adversaries of EMDT are malware
programs that break into a host through the network or
removable drives. As OS is the most popularly attractive to
hackers, the description of EMDT is designed Appling it to
operating systems with some changes.
The object and parameter signify the target and parameter of
4. Overview of EMDT the operation, respectively. Specific malware behaviours
monitored in the current version of EMDT, which includes
4.1 Overview the 30 critical malware behaviours shown in Table 1.
Moreover, EMDT allocates dynamic addition of new
The design of an access control mechanism is to define the behaviours. EMDT utilizes the subject label and behaviour
security label. We initiate a new form of security label called to build a decision while normal MACs use the subject label
suspicious label for our EMDT approach. It has two values: , object label,operation and parameter. As behaviour consists
suspicious and benign. Meanwhile, EMDT only allocates a of operation, object and parameter, EMDT actually uses the
suspicious label to a process or an executable, because a same four factors of normal MAC decision. Moreover,
process is possibly the agent of an intruder and an executable EMDT’s decision procedure produces three possible access
determines the execution flow of a procedure which control results: “allow,” “deny,” and “change label,” which
represents an intruder. When a process requests to access be similar to those of normal MACs. The detailed decision
these entities, EMDT mainly uses their DAC information to logic of Tracer is shown in Table 1. The detection and
make access control decisions, thus a vast amount of tracing actions guide to the decision result “change label,”
configuration work can be reduced while keeping traditional while restriction action leads to “deny.” All access requests
Volume 3 Issue 9, September 2014
www.ijsr.net
Paper ID: SEP14492 1780
Licensed Under Creative Commons Attribution CC BY
International Journal of Science and Research (IJSR)
ISSN (Online): 2319-7064
Impact Factor (2012): 3.358
not denied are allocated. As an online approach, Tracer be detection and MAC systems (Fraser, 2000; Wang et al.,
able to produce the FP rate lower than that of behaviour- 2008) that trace information flow on OS level, Tracer simply
blocking mechanisms in commercial antivirus software. This focuses on the tagging of executables while ignoring non
is attained as a MAC system, EMDT blocks a behaviour executables and directories. This is because an executable
based simultaneously on the behaviour and security label signifies the possible execution flow of the process loading
(i.e., the suspicious label of the current process), rather than it, thus it ought to be deemed as an inactive intruder while a
simplify the behaviour as done by a behaviour-blocking process is treated as an active intruder (Fig. 2). For tagging
system. processes, we observed that the excessive number of tags
mainly come from tracing Interposes Communication, i.e.,
4.2 Detecting intruders spotting a process as suspicious if it obtains IPC data from a
suspicious process. To address this issue, Tracer only tags a
The detecting action is liable for identifying all potential process receiving data from dangerous IPCs that can be
intruders. we design a light-weight intrusion detection exploited by a malware program to acquire control of the
algorithm that can identify all potential intruders but may process to make arbitrary malicious behaviours.
have a relatively higher FP rate at the initial step. Tracing
and restricting actions, will still agree to it to run rather than 4.4 Restricting Intruders
stop it immediately, but only avoid it from executing
featured malware behaviours. As depicted in the above Fig. In order to disable malware programs on a host, the
1, the detection works at two levels: entrance and interior: restricting action monitors and blocks intruders’ requests for
executing critical malware behaviours listed in Table 2. To
follow the principle of complete mediation for building a
security protection system, Tracer further restricts two
extensive
Where, D (P) is detection of process, signature s belongs to
signature based, it comes in distrustful folder. The detection
at entrance attempts to check all possible venues through
which a malware program may break into the system.
behaviours, called generic malware behaviours, to guard
security more widely. The first one is “Steal confidential
information,” which stands for all illegal reading of
Figure 2: The mechanism to dynamicaly detecting the confidential information from files and registry entries. The
malware behaviours to OS other is “Damage system integrity,” which be an illegal
modifications of the files and registry entries that need
4.3 Tracing Intruders preserving integrity. All behaviours restricted are listed on
the column “restrict” in Table 2. In summary, the restricting
To track intruders within an operating system, one can utilize action consists of three rules (Fig. 4):
OS-level information flow as done in King and Chen (2003)
and Goel et al. (2005). However, a main challenge for • Restricting critical malware behaviours
leveraging OS level information flow to trace suspicious • Restricting generic malware behaviours
entities is that, file and process tagging normally leads the • Restricting behaviours bypassing Tracer
entire system to be floated with “suspicious” labels and thus
earns too many FPs. To address this issue, we suggest the By mediating all these behaviours, Tracer is able to
following two methods to limit the number of tagged files safeguard system security and prevent a malware program
and processes in a single OS while averting malware from propagating itself in the system. To be exact,
programs from evading the tracing as much as possible. For confidentiality is mainly accomplished by blocking the
tagging files, unlike the approaches in King and Chen (2003) generic behaviour “Steal confidential information;”
and Goel et al. (2005) the schemes of several malware
Volume 3 Issue 9, September 2014
www.ijsr.net
Paper ID: SEP14492 1781
Licensed Under Creative Commons Attribution CC BY
International Journal of Science and Research (IJSR)
ISSN (Online): 2319-7064
Impact Factor (2012): 3.358
Then Copy the file into buffer Return (permit the File to
monitor)
Algorithm 2 is given below for detection that correlate read
and writes operations by comparing buffer contents are more
complicated to be circumvented than the other candidate
algorithms, e.g., comparing buffer addresses. In the worst
case that a malware program successfully circumvents the
algorithms, EMDT still can tail it by monitoring related
behaviours, e.g., “Create executables,” since file-copying
behaviours require to create executables.
Algorithm 2:
Detecting the Malware Process:
Figure 4: Dynamically restricting and detecting the malware Input: File to be read,
behaviors using EMDT process Buffer writer Process:
If (File ! = Copying Behaviour)||
integrity is generally protected by blocking the generic (Current Process = = suspicious)
behaviour “Damage system integrity;” availability is guarded Return Operation To Buffer
by blocking the behaviours listed in Table 2 with the capital For (Node of file = Read list of Buffer)
letter A attached. The algorithm 1 may impose a relatively If (File==Node)
elevated overhead only on the malware processes that Statement: Attach the File in the Buffer writer
frequently exhibit file copying behaviours but not on benign Else Statement: Blocking file from Corruption
processes and the suspected processes that are actually Then Copy the malware type into bufferwriter
gentle. Return (Malware type to buffer)
Algorithm 1: 4.5 Dynamic changes of malware behaviours detection
Monitoring the Application Process: process
Input: File to be read,
Buffer reader Process: If (File! = Copying Behaviour)|| EMDT can be able to dynamically add in new behaviours to
(Current Process = = Benign) monitor. Behaviour consists of object, operation and
Return Operation To Buffer parameter. For example, the operation create-file
For (Node of file = Read list of Buffer) corresponds to two system calls: NtOpenFile and
If (File = = Node) NtCreateFile. In contrast, a single system call might contain
Statement: Attach the File in the Buffer reader more than one operation. In each concerned system call, we
Else Statement: copy the File into Node (Stack) for set up one or more checkpoints, each of which is dependable
Blocking for checking the behaviours belonging to the same
Operation with the support of a modifiable behaviour list in that it can be initiated when the system restarts, finally
memory. listens at port 113 to accept commands from a remote
attacker. On a host without EMDT allowed, all above steps
5. Evaluation Results are successfully executed. However, after activating the
EMDT protection, the malware behaviour “Copy itself” is
Table 3 is given below explains the detailed test results of 5 blocked, i.e., the malware cannot generate a new copy of
selected malware samples. We can see that all the malware itself in the system folder. Consequently, the rest of the
samples are successfully disabled via the restriction of their behaviours do not emerge anymore because these behaviours
malware behaviours. For example, the worm “Worm.” depend on the new process launched from the malware’s
downloaded from the local website has the following main copy. In other words, the worm is disabled.
steps for function: it first copies itself, i.e., regsv.exe, to hard
drive in OS, then runs regsv.exe as a new process, the new
process then inserts a value under registry key regsv.exe so
Volume 3 Issue 9, September 2014
www.ijsr.net
Paper ID: SEP14492 1782
Licensed Under Creative Commons Attribution CC BY
International Journal of Science and Research (IJSR)
ISSN (Online): 2319-7064
Impact Factor (2012): 3.358
References
[1] Badger, L., D.F. Sterne, D.L. Sherman, K.M. Walker
and S.A. Haghighat, 1995. Practical domain and type
enforcement for UNIX. Proceeding of the IEEE
Symposium on Security and Privacy (S&P), pp: 66-77.
[2] Fraser, T., 2000. LOMAC: Low water-mark integrity
protection for COTS environments. Proceeding of the
IEEE Symposium on Security and Privacy (SP’ 00), pp:
230-245.
[3] Goel, A., K. Po, K. Farhadi, Z. Li and E. Lara, 2005.
The taser intrusion recovery system. Proceeding of the
20th ACM Symposium on Operating Systems Principles
(SOSP ’05), pp: 163-176.
[4] Kaspersky Lab, 2012. Retrieved from:
http://www.kaspersky.com/.
[5] King, S.T. and P.M. Chen, 2003. Backtracking
intrusions. Proceeding of the 19th ACM Symposium on
Operating Systems Principles (SOSP ’03), pp: 223-236.
[6] Kirda, E., C. Kruegel, V.G. Banks and R.A. Kemmerer,
2006. Behavior-based spyware detection. Proceeding of
the 15th Conference on USENIX Security Symposium
(USENIX-SS ’06).
[7] Li, N., Z. Mao and H. Chen, 2007. Usable mandatory
integrity protection for operating systems. Proceeding of
the IEEE Symposium on Security and Privacy (SP ’07),
pp: 164-178.
fundamental reason is that the antimalware tools identify a [8] Martignoni, L., E. Stinson, M. Fredrikson, S. Jha and
suspicious behaviour only supported on the behaviour itself J.C. Mitchell, 2008. A layered architecture for detecting
while Tracer further regard as the suspicious label of the malicious behaviors. Proceeding of the 11th
process requesting the behaviour (Fig. 5 and 6). International Symposium on Recent Advances in
Intrusion Detection, pp: 78-97.
[9] Microsoft, 2012. Mandatory Integrity Control. Retrieved
6. Conclusion
from:http://en.wikipedia.org/wiki/MandatoryIntegrityCo
ntrol.
In this study, we propose a novel MAC enforcement
[10] Nachenberg, C., 2002. Behaviors Blocking: The Next
approach that integrates intrusion detection and tracing to
Step in Anti-Virus Protection. Retrieved from:
guard against malware in a commercial OS. We have
http://www.securityfocus.com/infocus/1557.
extracted 30 critical malware behaviours and three common
[11] Shan, Z., X. Wang and T. Chiueh, 2011. Tracer:
malware characteristics for the incompatibility and low
Enforcing mandatory access control in commodity os
usability problems in MAC, which will benefit other
with the support of light-weight intrusion detection and
researchers in this area. Based on these studies, we propose a
tracing. Proceeding of the 6th ACM Symposium on
novel MAC enforcement approach, called EMDT using
Information, Computer and Communication Security,
Hidden markov model, to disable malware timely without
pp: 135-144.
need of malware signatures or other knowledge in progress.
[12] Viper Inc., 2012. Retrieved from: http://www. vipre.
The novelty of Tracer design is two- fold. One is to utilize
com/vipre/, 2012.
intrusion detection and tracing to regularly configure
[13] Wang, X., Z. Li, J.Y. Choi and N. Li, 2008. PRECIP:
security labels. EMDT system doesn’t restrict the suspected
Towards practical and retrofittable confidential
intruders right away but permits them to run as long as
information protection. Proceeding of the 15th Network
feasible except blocking their critical malware behaviours.
and Distributed System Security Symposium.
This design generates a MAC system with good
compatibility and usability. We have implemented Tracer in
several OS and the evaluation results show that it can Author Profile
successfully guard against a set of real-world malware
Dokuparthi Prasanthi received the MCA degree from Acharya
programs, including unknown malware programs, with much
Nagarjuna University Vijayawada in 2012 and pursuing M.tech
small FP rate than that of commercial antimalware degree in Computer science and Engineering from Anurag Group
techniques. In future we are going to initiate this study for a of Institutions (Formerly CVSR College of Engineering) JNTU
large web server runs the application front-end logic and Hyderabad, India.
data are outsourced to a database or file server where there is
increase in application and data complexity. V. Rama Krishna working as Assistant Professor in Computer
Science and Engineering from Anurag Group of Institutions
(Formerly CVSR College of Engineering) JNTU Hyderabad, India.
Volume 3 Issue 9, September 2014
www.ijsr.net
Paper ID: SEP14492 1783
Licensed Under Creative Commons Attribution CC BY