DOKK Library

Malware Detection and Tracer Approach for Operating System

Authors Dokuparthi Prasanthi V. Rama Krishna

License CC-BY

                                  International Journal of Science and Research (IJSR)
                                                       ISSN (Online): 2319-7064
                                                      Impact Factor (2012): 3.358

          Malware Detection and Tracer Approach for
                     Operating System
                                            Dokuparthi Prasanthi1, V. Rama Krishna2
                              M.Tech student, Department of CSE, Anurag Group of Institutions, Hyderabad, India
                          Assistant Professor, Department of CSE, Anurag Group of Institutions, Hyderabad, India

Abstract: Modern computer systems are built on a foundation of software components from a variety of vendors. While critical
applications might undergo extensive testing and evaluation procedures, the heterogeneity of software sources hazardous the integrity of
the execution environment for these trusted programs. For instance, if an attacker can be able to merge an application exploit with
privilege increase vulnerability, the Operating System (OS) can become corrupted. Mandatory Access Control (MAC) in a commercial
operating system to handle malware problem is a challenge but also a capable approach. The firmest barriers to apply MAC to defeat
malware programs are the incompatible and unusable problems in existing MAC systems. The aim of our study is to address these issues
design a novel Efficient Malware Detection and Tracer design (EMDT) using Hidden Markov model, which incorporates intrusion
detection and tracing in an operating system. In this proposed approach conceptually consists of three actions: tracing, detecting and
restricting deduced intruders .The novelty of the proposed study is that it leverages light-weight intrusion detection and tracing
techniques to automate security label configuration that is widely acknowledged as a tough issue when applying a MAC system in
practice. The other is that, rather than controlling information flow as a traditional MAC does, it traces intruders and restricts only their
significant malware behaviours’, where intruders characterize processes and executables that are potential agents of a remote attacker.
Our prototyping and testing’s on Windows operating system show that Tracer can effectively defeat all malware samples tested via
blocking malware behaviours while not causing a significant compatibility problem.

Keywords: Detection, intrusion, malware, tracing, vulnerability.

1. Introduction                                                         Each process or executable has two states, suspicious or
                                                                        benign. The contributions of this study are We introduce
Malicious software (i.e., Malware) is one of the most severe            EMDT, a novel MAC enforcement approach which
computer security problems today. A network of hosts which              combines intrusion detection and tracing techniques to
are cooperated by malware and controlled by attackers can               disable malware on a commercial OS in a compatible and
cause a lot of damages to information systems. As a useful              usable manner. We have implemented EMDT to immobilize
malware defence technology, Mandatory Access Control                    malware timely without need of malware signatures. We
(MAC) works without relying on malware signatures and                   investigate the root reason so discover compatibility and low
blocks malware behaviours before they cause security                    usability problems of existing MACs. Although not all the
damage. Even if an unauthorized user manages to breach                  observations are brand new, we consider that understanding
other layers of defence, MAC is capable of act as the last              these reasons more comprehensively and illustrating them
shelter to avoid the entire host from being compromised.                through the design of an actual system are useful for other
However, MAC mechanisms built in commercial operating                   MAC researchers.
systems (OS) often go through from two problems which
make general users unenthusiastic to assume them. One                   2. Related Work
problem is that a built-in MAC is mismatched with a lot of
application software and thus interferes with their running             DTE proposed by Badger et al. (1995) is a classical MAC
and the other problem is low usability, which creates it                model to confine process execution, which group’s processes
difficult to configure MAC properly. Our observations are as            and files into domains and types, respectively and controls
follows: The incompatibility problem is introduced because              accesses among domains and types. Tracer can be
the security labels of existing MACs are not capable to                 considered as a simplified DTE that has two domains (i.e.,
distinguish between malicious and benign entities, which                benign and suspicious) and four types (i.e., benign, read-
Causes a enormous number of False Positives (FP) (i.e.,                 protected, write-protected and suspicious). Moreover, Tracer
treating benign operations as malicious) thus avoiding many             can usually configure the DTE attributes (i.e., domain and
benign software from performing legal operations; the low-              type) of processes and files under the maintain of intrusion
usability problem is launched, because existing MACs are                detection and tracing so as to develop usability. PRECIP
not capable to automatically label the huge number of                   Wang et al. (2008) addresses several practical issues that are
entities in OS and thus require tough configuration work at             significant to contain spyware that be determined to leak
End users. With these investigation results, our main                   sensitive information. The risk-adaptive access controls
objective is to propose a novel MAC enforcement approach                (Kaspersky Lab, 2012). That targets to create access control
EMDT, this consists of three actions: Detection, tracing and            more dynamic so as to attain a better tradeoff between risk
restriction.                                                            and benefit. Most existing antimalware technologies are
                                                                        based on detection (Kirda et al., 2006; Martignoni et al.,
                                                                        2008). Tracer tries to combine detection and access control
                                               Volume 3 Issue 9, September 2014
      Paper ID: SEP14492                                                                                                       1779
                                        Licensed Under Creative Commons Attribution CC BY
                              International Journal of Science and Research (IJSR)
                                                   ISSN (Online): 2319-7064
                                                  Impact Factor (2012): 3.358
so that it not only can detect but also can block malware          usage conventions unchanged.
behaviours before their harming security. Antimalware
technology that be similar to Proposed EMDT is behaviour
blocking (Nachenberg, 2002) which can confine the
behaviours of definite adverse programs that are profiled in
advance. Many commercial antimalware tools (Kaspersky
Lab, 2012; Viper Inc., 2012) also have a behaviour-based
module to protect against unknown malware programs.

Problems in MAC

Incompatibility is a familiar problem when enforcing a MAC
modeling operating system (Li et al., 2007; Fraser, 2000;
Wang et al., 2008). To examine its core reason, in a secure
network environment, we set up two mechanisms to run
MAC enforced operating systems with MLS policy allowed
and MAC module allowed. After a few days, we surveyed
that these MAC systems produced a enormous number of log                             Figure 1: EMDT Overview
records about denied accesses, which specified that some
applications failed and some acted irregularly. As the             The above Fig. 1 gives an overview of EMDT which
operation environment is secure without intrusion and              consists of three types of actions, tracing, detection and
malware, these denied accesses are thus “false positive.”          restriction. Each process or executable has two states,
However, from the view of intrusion thwarting, these               suspicious and benign. The restriction action forbids a
processes do not necessarily represent intruders so that their     suspected intruder to make malware behaviours in order to
“read” or “write” accesses to the/tmp should not be merely         guard CIAP. That is to protect integrity, confidentiality and
denied. Although it is possible to resolve this problem by         availability, as well as to stop malware propagation. The
adding “hiding sub directories” under/tmp, it is still difficult   three actions study as follows: Once detecting a suspected
to eliminate the FPs resulting from many other shared              process or executable, EMDT labels it as suspicious and
entities on an OS Relying on these labels, a MAC system            traces its descendent and interacted processes, as well as its
habitually fails to make correct decisions on intrusion            generated executables. EMDT does not restrict benign
blocking which eventually results in many FPs. Low                 processes at all and permits suspicious processes to run as
usability is another problem in a MAC-enabled system, as it        long as possible but stops their malware behaviours that
often requires make difficult configurations and                   would cause security damages.
unconventional ways of usage.

3. Proposed System
Efficient Malware Detection and Tracer (EMDT): In this
section, we present our EMDT approach that aims to
immobilize malware in a OS by disagreeing malware
behaviours. The adversaries of EMDT are malware
programs that break into a host through the network or
removable drives. As OS is the most popularly attractive to
hackers, the description of EMDT is designed Appling it to
operating systems with some changes.
                                                                   The object and parameter signify the target and parameter of
4. Overview of EMDT                                                the operation, respectively. Specific malware behaviours
                                                                   monitored in the current version of EMDT, which includes
4.1 Overview                                                       the 30 critical malware behaviours shown in Table 1.
                                                                   Moreover, EMDT allocates dynamic addition of new
The design of an access control mechanism is to define the         behaviours. EMDT utilizes the subject label and behaviour
security label. We initiate a new form of security label called    to build a decision while normal MACs use the subject label
suspicious label for our EMDT approach. It has two values:         , object label,operation and parameter. As behaviour consists
suspicious and benign. Meanwhile, EMDT only allocates a            of operation, object and parameter, EMDT actually uses the
suspicious label to a process or an executable, because a          same four factors of normal MAC decision. Moreover,
process is possibly the agent of an intruder and an executable     EMDT’s decision procedure produces three possible access
determines the execution flow of a procedure which                 control results: “allow,” “deny,” and “change label,” which
represents an intruder. When a process requests to access          be similar to those of normal MACs. The detailed decision
these entities, EMDT mainly uses their DAC information to          logic of Tracer is shown in Table 1. The detection and
make access control decisions, thus a vast amount of               tracing actions guide to the decision result “change label,”
configuration work can be reduced while keeping traditional        while restriction action leads to “deny.” All access requests

                                           Volume 3 Issue 9, September 2014
      Paper ID: SEP14492                                                                                            1780
                                    Licensed Under Creative Commons Attribution CC BY
                               International Journal of Science and Research (IJSR)
                                                    ISSN (Online): 2319-7064
                                                   Impact Factor (2012): 3.358
not denied are allocated. As an online approach, Tracer be           detection and MAC systems (Fraser, 2000; Wang et al.,
able to produce the FP rate lower than that of behaviour-            2008) that trace information flow on OS level, Tracer simply
blocking mechanisms in commercial antivirus software. This           focuses on the tagging of executables while ignoring non
is attained as a MAC system, EMDT blocks a behaviour                 executables and directories. This is because an executable
based simultaneously on the behaviour and security label             signifies the possible execution flow of the process loading
(i.e., the suspicious label of the current process), rather than     it, thus it ought to be deemed as an inactive intruder while a
simplify the behaviour as done by a behaviour-blocking               process is treated as an active intruder (Fig. 2). For tagging
system.                                                              processes, we observed that the excessive number of tags
                                                                     mainly come from tracing Interposes Communication, i.e.,
4.2 Detecting intruders                                              spotting a process as suspicious if it obtains IPC data from a
                                                                     suspicious process. To address this issue, Tracer only tags a
The detecting action is liable for identifying all potential         process receiving data from dangerous IPCs that can be
intruders. we design a light-weight intrusion detection              exploited by a malware program to acquire control of the
algorithm that can identify all potential intruders but may          process to make arbitrary malicious behaviours.
have a relatively higher FP rate at the initial step. Tracing
and restricting actions, will still agree to it to run rather than   4.4 Restricting Intruders
stop it immediately, but only avoid it from executing
featured malware behaviours. As depicted in the above Fig.           In order to disable malware programs on a host, the
1, the detection works at two levels: entrance and interior:         restricting action monitors and blocks intruders’ requests for
                                                                     executing critical malware behaviours listed in Table 2. To
                                                                     follow the principle of complete mediation for building a
                                                                     security protection system, Tracer further restricts two
Where, D (P) is detection of process, signature s belongs to
signature based, it comes in distrustful folder. The detection
at entrance attempts to check all possible venues through
which a malware program may break into the system.

                                                                     behaviours, called generic malware behaviours, to guard
                                                                     security more widely. The first one is “Steal confidential
                                                                     information,” which stands for all illegal reading of
Figure 2: The mechanism to dynamicaly detecting the                  confidential information from files and registry entries. The
malware behaviours to OS                                             other is “Damage system integrity,” which be an illegal
                                                                     modifications of the files and registry entries that need
4.3 Tracing Intruders                                                preserving integrity. All behaviours restricted are listed on
                                                                     the column “restrict” in Table 2. In summary, the restricting
To track intruders within an operating system, one can utilize       action consists of three rules (Fig. 4):
OS-level information flow as done in King and Chen (2003)
and Goel et al. (2005). However, a main challenge for                • Restricting critical malware behaviours
leveraging OS level information flow to trace suspicious             • Restricting generic malware behaviours
entities is that, file and process tagging normally leads the        • Restricting behaviours bypassing Tracer
entire system to be floated with “suspicious” labels and thus
earns too many FPs. To address this issue, we suggest the            By mediating all these behaviours, Tracer is able to
following two methods to limit the number of tagged files            safeguard system security and prevent a malware program
and processes in a single OS while averting malware                  from propagating itself in the system. To be exact,
programs from evading the tracing as much as possible. For           confidentiality is mainly accomplished by blocking the
tagging files, unlike the approaches in King and Chen (2003)         generic behaviour “Steal confidential information;”
and Goel et al. (2005) the schemes of several malware

                                            Volume 3 Issue 9, September 2014
      Paper ID: SEP14492                                                                                              1781
                                     Licensed Under Creative Commons Attribution CC BY
                                International Journal of Science and Research (IJSR)
                                                      ISSN (Online): 2319-7064
                                                     Impact Factor (2012): 3.358
                                                                       Then Copy the file into buffer Return (permit the File to

                                                                       Algorithm 2 is given below for detection that correlate read
                                                                       and writes operations by comparing buffer contents are more
                                                                       complicated to be circumvented than the other candidate
                                                                       algorithms, e.g., comparing buffer addresses. In the worst
                                                                       case that a malware program successfully circumvents the
                                                                       algorithms, EMDT still can tail it by monitoring related
                                                                       behaviours, e.g., “Create executables,” since file-copying
                                                                       behaviours require to create executables.

                                                                       Algorithm 2:
                                                                       Detecting the Malware Process:
Figure 4: Dynamically restricting and detecting the malware             Input: File to be read,
             behaviors using EMDT process                              Buffer writer Process:
                                                                       If (File ! = Copying Behaviour)||
integrity is generally protected by blocking the generic               (Current Process = = suspicious)
behaviour “Damage system integrity;” availability is guarded            Return Operation To Buffer
by blocking the behaviours listed in Table 2 with the capital           For (Node of file = Read list of Buffer)
letter A attached. The algorithm 1 may impose a relatively             If (File==Node)
elevated overhead only on the malware processes that                   Statement: Attach the File in the Buffer writer
frequently exhibit file copying behaviours but not on benign            Else Statement: Blocking file from Corruption
processes and the suspected processes that are actually                Then Copy the malware type into bufferwriter
gentle.                                                                Return (Malware type to buffer)

Algorithm 1:                                                           4.5 Dynamic changes of malware behaviours detection
Monitoring the Application Process:                                    process
 Input: File to be read,
Buffer reader Process: If (File! = Copying Behaviour)||                EMDT can be able to dynamically add in new behaviours to
(Current Process = = Benign)                                           monitor. Behaviour consists of object, operation and
Return Operation To Buffer                                             parameter. For example, the operation create-file
For (Node of file = Read list of Buffer)                               corresponds to two system calls: NtOpenFile and
 If (File = = Node)                                                    NtCreateFile. In contrast, a single system call might contain
Statement: Attach the File in the Buffer reader                        more than one operation. In each concerned system call, we
 Else Statement: copy the File into Node (Stack) for                   set up one or more checkpoints, each of which is dependable
Blocking                                                               for checking the behaviours belonging to the same

Operation with the support of a modifiable behaviour list in           that it can be initiated when the system restarts, finally
memory.                                                                listens at port 113 to accept commands from a remote
                                                                       attacker. On a host without EMDT allowed, all above steps
5. Evaluation Results                                                  are successfully executed. However, after activating the
                                                                       EMDT protection, the malware behaviour “Copy itself” is
Table 3 is given below explains the detailed test results of 5         blocked, i.e., the malware cannot generate a new copy of
selected malware samples. We can see that all the malware              itself in the system folder. Consequently, the rest of the
samples are successfully disabled via the restriction of their         behaviours do not emerge anymore because these behaviours
malware behaviours. For example, the worm “Worm.”                      depend on the new process launched from the malware’s
downloaded from the local website has the following main               copy. In other words, the worm is disabled.
steps for function: it first copies itself, i.e., regsv.exe, to hard
drive in OS, then runs regsv.exe as a new process, the new
process then inserts a value under registry key regsv.exe so

                                             Volume 3 Issue 9, September 2014
      Paper ID: SEP14492                                                                                                 1782
                                       Licensed Under Creative Commons Attribution CC BY
                             International Journal of Science and Research (IJSR)
                                                  ISSN (Online): 2319-7064
                                                 Impact Factor (2012): 3.358
                                                                  [1] Badger, L., D.F. Sterne, D.L. Sherman, K.M. Walker
                                                                       and S.A. Haghighat, 1995. Practical domain and type
                                                                       enforcement for UNIX. Proceeding of the IEEE
                                                                       Symposium on Security and Privacy (S&P), pp: 66-77.
                                                                  [2] Fraser, T., 2000. LOMAC: Low water-mark integrity
                                                                       protection for COTS environments. Proceeding of the
                                                                       IEEE Symposium on Security and Privacy (SP’ 00), pp:
                                                                  [3] Goel, A., K. Po, K. Farhadi, Z. Li and E. Lara, 2005.
                                                                       The taser intrusion recovery system. Proceeding of the
                                                                       20th ACM Symposium on Operating Systems Principles
                                                                       (SOSP ’05), pp: 163-176.
                                                                  [4] Kaspersky         Lab,      2012.      Retrieved     from:
                                                                  [5] King, S.T. and P.M. Chen, 2003. Backtracking
                                                                       intrusions. Proceeding of the 19th ACM Symposium on
                                                                       Operating Systems Principles (SOSP ’03), pp: 223-236.
                                                                  [6] Kirda, E., C. Kruegel, V.G. Banks and R.A. Kemmerer,
                                                                       2006. Behavior-based spyware detection. Proceeding of
                                                                       the 15th Conference on USENIX Security Symposium
                                                                       (USENIX-SS ’06).
                                                                  [7] Li, N., Z. Mao and H. Chen, 2007. Usable mandatory
                                                                       integrity protection for operating systems. Proceeding of
                                                                       the IEEE Symposium on Security and Privacy (SP ’07),
                                                                       pp: 164-178.
fundamental reason is that the antimalware tools identify a       [8] Martignoni, L., E. Stinson, M. Fredrikson, S. Jha and
suspicious behaviour only supported on the behaviour itself            J.C. Mitchell, 2008. A layered architecture for detecting
while Tracer further regard as the suspicious label of the             malicious behaviors. Proceeding of the 11th
process requesting the behaviour (Fig. 5 and 6).                       International Symposium on Recent Advances in
                                                                       Intrusion Detection, pp: 78-97.
                                                                  [9] Microsoft, 2012. Mandatory Integrity Control. Retrieved
6. Conclusion
In this study, we propose a novel MAC enforcement
                                                                  [10] Nachenberg, C., 2002. Behaviors Blocking: The Next
approach that integrates intrusion detection and tracing to
                                                                       Step in Anti-Virus Protection. Retrieved from:
guard against malware in a commercial OS. We have
extracted 30 critical malware behaviours and three common
                                                                  [11] Shan, Z., X. Wang and T. Chiueh, 2011. Tracer:
malware characteristics for the incompatibility and low
                                                                       Enforcing mandatory access control in commodity os
usability problems in MAC, which will benefit other
                                                                       with the support of light-weight intrusion detection and
researchers in this area. Based on these studies, we propose a
                                                                       tracing. Proceeding of the 6th ACM Symposium on
novel MAC enforcement approach, called EMDT using
                                                                       Information, Computer and Communication Security,
Hidden markov model, to disable malware timely without
                                                                       pp: 135-144.
need of malware signatures or other knowledge in progress.
                                                                  [12] Viper Inc., 2012. Retrieved from: http://www. vipre.
The novelty of Tracer design is two- fold. One is to utilize
                                                                       com/vipre/, 2012.
intrusion detection and tracing to regularly configure
                                                                  [13] Wang, X., Z. Li, J.Y. Choi and N. Li, 2008. PRECIP:
security labels. EMDT system doesn’t restrict the suspected
                                                                       Towards practical and retrofittable confidential
intruders right away but permits them to run as long as
                                                                       information protection. Proceeding of the 15th Network
feasible except blocking their critical malware behaviours.
                                                                       and Distributed System Security Symposium.
This design generates a MAC system with good
compatibility and usability. We have implemented Tracer in
several OS and the evaluation results show that it can            Author Profile
successfully guard against a set of real-world malware
                                                                  Dokuparthi Prasanthi received the MCA degree from Acharya
programs, including unknown malware programs, with much
                                                                  Nagarjuna University Vijayawada in 2012 and pursuing
small FP rate than that of commercial antimalware                 degree in Computer science and Engineering from Anurag Group
techniques. In future we are going to initiate this study for a   of Institutions (Formerly CVSR College of Engineering) JNTU
large web server runs the application front-end logic and         Hyderabad, India.
data are outsourced to a database or file server where there is
increase in application and data complexity.                      V. Rama Krishna working as Assistant Professor in Computer
                                                                  Science and Engineering from Anurag Group of Institutions
                                                                  (Formerly CVSR College of Engineering) JNTU Hyderabad, India.

                                          Volume 3 Issue 9, September 2014
      Paper ID: SEP14492                                                                                           1783
                                    Licensed Under Creative Commons Attribution CC BY