BYU logo Computer Science
CS 465 Introduction to Security and Privacy

How malware works

Ungraded Quiz

  • Describe some of the ways a virus can hide itself from detection
  • How does a worm spread?
  • What is the purpose of a rootkit?
  • What is hooking?
  • How does inline hooking work?

Key concepts

Anti-detection strategies

  • see Figure 7.2, page 191

  • virus with encrypted body, note the key is in the decryptor section

  • polymorphic virus changes the decryptor portion across infections, making detection harder (the signature changes)

    • note how an anti-virus tool will run a program in an emulated environment first, then check the decrypted body for a signature, since that is static
    • use all the tricks you can to fight malware
  • virus with an external decryption key — makes manual analysis harder

    • attackers will use all the tricks they can to fight back
  • metamorphic virus — entire virus evolves per instance, not encrypted

    • rewrites its own code!!
  • can also try to hide signs like file sizes and timestamps, the fact that a process is running, etc.

  • importance of reverse engineering — see text on same page

  • understand what an auto-rooter does

    • program that makes it easy to take over a machine with a point and click interface
    • attackers are not necessarily skilled, but may be using tools others have built

How worms spread

  • scanning IP address ranges (may be random)
  • both IPv4 and IPv6
  • may prioritize local addresses first (like the Code Red II Worm)
  • may use context local to that machine, such as URLs on the disk, addresses of websites you regularly visit, email addresses, etc.
  • how to succeed as a worm (spread faster)
    • hit-list scanning — collect vulnerable targets before infecting them
    • permutation scanning — each new infection scans addresses in a different order, so overall the worm finds new victims quickly
    • Internet-scale hit lists — can collect a list of all web servers of a particular version in advance
  • combining techniques means a worm can infect all vulnerable machines on the Internet in tens of seconds

The Morris Worm

  • 1988
  • 10% of Internet devices
  • traffic overloaded parts of the network and caused denial of service
  • no malicious payload!
  • see list of exploits, bottom of page 193

Stealth

  • Trojan horse — program appears to be benign and then also does something malicious — user installs it willingly
  • backdoor — a way to bypass normal access controls, typically hiding in otherwise benevolent code

Rootkits

  • a rootkit — hides its presence and facilitates other malicious code
    • user mode rootkit provides root access
    • kernel mode rootkit provides kernel mode access (more powerful and harder to detect/remove)
    • understand superuser vs kernel mode or supervisor mode — see discussion on page 195
  • stealth: “replace system code, modify system data structures that do not impact core OS functions, alter/erase log files, and filter results reported back to processes”
  • keyloggers
  • surveillance
  • hijacking system calls — see Figure 7.4, page 197
    • note that hooking is used legitimately, e.g. by anti-virus programs or to extend OS functionality
  • inline hooking — see FIgure 7.5, page 198
    • see also inline hooking on Windows
    • the detour lets malware modify the arguments to the system call
    • the trampoline lets malware modify the return value of the system call
  • hiding a process, files, or open network connections
    • modify kernel data structures
    • remove results of a system call (see inline hooking above)
  • understand there are loadable kernel modules that can be used to create a rootkit