Archive

Archive for the ‘OS Internals’ Category

Dump Windows password hashes efficiently

January 29, 2012 Leave a comment
Categories: OS Internals, Tech

Read or Update a registry key for ALL users on a system

January 13, 2012 115 comments

Have you ever needed to read or update a registry key that is stored in each user’s HKEY_CURRENT_USER or HKEY_CLASSES_ROOT hive? Have you also ever needed to read or update it for ALL users on the system, as well as make it the default setting when a new user profile is created?

That can be a bit of a daunting task. One solution is to add the registry key update to the user’s logon script.

UPDATE: As a commenter pointed out, an alternative solution is to use the built-in “Active Setup” functionality which can run a script the next time each user logs in, under that user’s context. In most scenarios, I believe this will be the best method for updating registry settings on a per-user basis.

What is Active-Setup? http://sccmpackager.blogspot.com/2013/07/active-setup-what-is-active-setup.html

However, the technique described in this post is very useful for when you need to read from each user’s registry settings (or write to) immediately and cannot wait for the next time a user logs onto the system.

Fortunately, there is another way that will immediately update all profiles (including the DEFAULT profile) and I wrote a vbscript to make it easier.

The source code (vbscript) is available here: https://github.com/MicksMix/RegUpdateAllUsers

CHANGELOG

  • Nov 15, 2013 – Able to update NTUSER.DAT and/or USRCLASS.DAT (HKCU and/or HKCR)
  • Aug 25, 2013 – Added ability to delete keys
  • Apr 23, 2013 – Added ability to write REG_BINARY values
  • Apr 11, 2013 – Fixed bug where it wouldn’t work when run by SYSTEM account
  • Mar 28, 2013 – Huge code cleanup and bug fixes
  • Jan 13, 2012 – Initial release

The script can  set REG_BINARY keys as long as they are in the format used by a regedit.exe export. For example:

[HKEY_CURRENT_USER\Software\_Test\MyTestBinarySubkey]
"My Test Binary Value"=hex:23,00,41,00,43,00,42,00,6c,00

To set this binary value using the script, you would modify line 82 to be:
SetBinaryRegKeys sRegistryRootToUse, strRegPathParent03, “My Test Binary Value”,“hex:23,00,41,00,43,00,42,00,6c,00”

The script works correctly even when run under the SYSTEM account.

The general way this script works:

  1. Update the currently logged on user’s HKCU (that’s easy enough)
  2. Then you must enumerate every profile on the system
  3. Find their ntuser.dat file (ntuser.dat contains the contents of the user’s HKCU hive)
  4. Find their usrclass.dat file (usrclass.dat contains the user’s HKCR hive)
  5. Load ntuser.dat and/or usrclass.dat into a temporary key in the HKLM hive (programmatically or using reg.exe)
  6. I use ‘HKLM\TempHive’ as the temporary key
  7. Then when you write to “HKLM\TempHive”you are actually editing that user’s HKCU hive.
  8. If you load ntuser.dat/usrclass.dat for the “Default” user, the settings will take effect for any NEW user profile created on the system
  9. If more than 1 user is currently logged on, you can edit their HKCU/HKCR hive by looking the user up by their SID under HKEY_USERS and writing to it at that location.

It’s a bit of a tedious job, so I wrote a VBScript that takes care of all of the steps listed above. This script has been tested on Windows XP and Windows 7 (x64), but should work on Windows 2000 and newer. It relies on “reg.exe” which ships with all versions of Windows.

Advertisement

The Windows “App Paths” Registry Key

August 19, 2011 Leave a comment

This was news to me. Basically, not every app has to be on your PATH to be launched via executable name only (from anywhere).

For example, on my x64 Windows 7 machine, WordPad is located at:

<C:\Program Files\Windows NT\Accessories\wordpad.exe>

This is *not* set in my PATH environment variable. But I can go to Start > Run, and type wordpad, and it launches.

Why does this work? Because of “App Paths”.

<HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\App Paths>

http://helgeklein.com/blog/2010/08/how-the-app-paths-registry-key-makes-windows-both-faster-and-safer/

Why can you start Mozilla Firefox by typing “firefox” in the Run dialog and press enter? Firefox.exe is not located in any directory in the path. The same with Outlook (type “outlook”), PowerShell (“powershell”), VMware Workstation (“vmware”) or Adobe Reader (“acrord32″). This “magic application starting thingy” works because of a little-known Windows feature based on the “App Paths” registry key.

Windows API developer Raymond Chen has more detail on the feature and history on why “App Paths” registry key was created in the first place.

http://blogs.msdn.com/b/oldnewthing/archive/2011/07/25/10189298.aspx

Also note that malware could use this as well. Raymond Chen clarifies it well:

Now, the intent was that the registered full path to the application is the same as the registered short name, just with a full path in front. For example, wordpad.exe registers the full path of %ProgramFiles%\Windows NT\Accessories\WORDPAD.EXE.

But there’s no check that the two file names match. The Pbrush folks took advantage of this by registering an application path entry for pbrush.exe with a full path of %SystemRoot%\System32\mspaint.exe: That way, when somebody types pbrush into the Run dialog, they get redirected to mspaint.exe.

Categories: OS Internals, Tech, Windows

Mac OS X auto-run locations

June 29, 2011 2 comments

The smart people over at Malicious Streams have created a python script to show applications set to auto-start on your Mac at either boot or user login. I have also pasted the current version (0.6a) of the script below:

#!/usr/bin/python
"""

    Description: osxautoruns.py - Lists applications set to auto-launch either during os startup or during login
    Platform:    Mac OS X, Python 2.x
    Copywrite:   (C)2010 Joel Yonts, Malicious Streams

    This program is free software: you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
    the Free Software Foundation, either version 3 of the License, or
    (at your option) any later version.

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details.

    See companion file, Licenses.txt, for a copy of the GNU General Public License
    or a copy is available at <http://www.gnu.org/licenses/>.

"""
__version__ = '0.6a'

import plistlib
import sys
import os
import operator
#import config
import Foundation

# Section: CONFIGURATION -------------------------------------------
PLIST_EXT=".plist"
USERDIR="/Users"

STARTUP_PLISTS=["/System/Library/LaunchDaemons",
                    "/Library/LaunchDaemons",
                    "/System/Library/LaunchAgents",
                    "/Library/LaunchAgents",
                    "/Library/StartupItems",
                    "/Library/Preferences/loginwindow.plist",
                    "~/Library/LaunchAgents",
                    "~/Library/Preferences/loginitems.plist",
                    "~/Library/Preferences/loginwindow.plist"]

LOGIN_PLIST="/Library/Preferences/loginwindow.plist"

USER_CONFIG_DIR="/private/var/db/dslocal/nodes/Default/users"
USER_DIR_BLACKLIST=["/var/empty", "/dev/null"]
USER_SHELL_BLACKLIST=["/usr/bin/false"]

# Section: PRIMARY CLASS DEFINITION -------------------------------------------
class AutoRuns:
	
	def __init__(self, verbose, user, mntPoint=""):


		self.target_user = user
				
		if mntPoint == "":
			self.live_response = True
			print "Analysis Type: Live Analysis"
		else:
			self.live_response = False
			print "Analysis Type: Mounted System"
			
		if verbose == True:
			self.verbose = True
		else:
			self.verbose = False
		
		# Build Comphrensive List of Startup Plist Paths
		self.startup_plists =STARTUP_PLISTS
		
		# Retrieve Full User if using SUDO perms, otherwise just get current user
		self.user_list=self._getUserList(mntPoint)
			
		self._expandUserPaths(mntPoint)
		self._expandPlistDirs(mntPoint)
	
		self.startupItems=self._parsePlists()

		self.startup_plists.sort()
		self.startupItems.sort(key=operator.itemgetter('User'))

	def _userExists(self, username):
		""" Determines if a specified user exists on the system.
		
		@type raw_data: string
		@param row_data: Username to be checked for existence
		
		@return: boolean
		"""
		userExists = False
		
		for user, homdir in self.user_list:
			if username == user:
				userExists =True
				
		return userExists
	
	def _getCurrentUser(self):
		""" Return current user name and home directory
		
		@type raw_data: string
		@param row_data: Mountpoint (if any) for OS X image.  Default is /
		
		@return: list containing name,home directory for each valid user
		"""
		try:
			logname = os.environ.get('LOGNAME')
			homedir = os.environ.get('HOME')
			return [[logname, homedir]]
		except:
			return []
	
	def _getUserList(self, mntPoint):
		""" Build a valid user list by reading configuration file(s).  
		Function also uses black lists to remove system accounts.
		
		@type raw_data: string
		@param row_data: Mountpoint (if any) for OS X image.  Default is /
		
		@return: list containing name,home directory for each valid user
		"""
		error_state = False
		error_msgs =[]

		user_list =[]
		if mntPoint!= "" and mntPoint.endswith('/')==False:
			mntPoint=mntPoint+os.sep
			
		user_config_dir=mntPoint+USER_CONFIG_DIR
		if os.access(user_config_dir, os.R_OK)==True:
			for user_config in os.listdir(user_config_dir):
				
				user_config_plist=self._readRawPlist(user_config_dir+os.sep+user_config)
				if user_config_plist != None:
					#print user_config_plist
					if user_config_plist.has_key("name") == True and user_config_plist.has_key("home") == True and user_config_plist.has_key("shell"):
						username=user_config_plist['name'][0]
						homedir=user_config_plist['home'][0]
						shell=user_config_plist['shell'][0]
						if USER_DIR_BLACKLIST.count(homedir)==0 and USER_SHELL_BLACKLIST.count(shell)==0:
							user_list.append([username, homedir])
				else:
					if self.verbose == True:
						error_msgs.append("Error reading: %s%s%s" % (user_config_dir, os.sep, user_config))
						
					error_state = True
					
		else:
			if self.verbose == True:
				error_msgs.append("Error accessing: "+str(user_config_dir))
				
			error_state = True
					
		if error_state == True and self.live_response == True:
			user_list =self._getCurrentUser()
			if user_list[0][0] != self.target_user:
				if self.verbose == True:
					print "===> WARNING: Error accessing user configurations:"
					for msg in error_msgs:
						print "\t"+msg
				else:
					print "===> WARNING: Error accessing user configurations (Use -V option for details)."		
						
				print "===> WARNING: Only global (SYSTEM Level) and current user startup items will be displayed."
				print "===> WARNING: Use of elevated (SUDO) permissions may provide access to startup items for all users.\n\n"
				


					
		if error_state == True and self.live_response == False:
			if self.verbose == True:
				print "===> WARNING: Error accessing user configurations:"
				for msg in error_msgs:
					print "\t"+msg
			else:
				print "===> WARNING: Error accessing user configurations (Use -V option for details)."	
			print "===> WARNING: Only global (SYSTEM Level) startup items will be displayed."
			print "===> WARNING: Use of elevated (SUDO) permissions may provide access to user startup items.\n\n"

		return user_list
		
		# --- Section: Parse Plists and Build Startup Items List -------
	def _OLD_readPlist(self, filename):  # Does not support binary plists
		""" Read and Parse the specified plist.  
		This function is currently not used because the libraries inability to deal with all plist types (Binary Plists).
		
		@type raw_data: string
		@param row_data: Full path to plist file
		
		@return: list containing plist data
		"""

		plist=plistlib.readPlist(filename)
		return plist
	
	def _readRawPlist(self, plistFile):
		""" Read and Parse the specified plist file.  
		This function uses the Foundation class to achieve compatibility with all plist types (This does break cross-platform compatibility).
		
		@type raw_data: string
		@param row_data: Full path to plist file
		
		@return: dictionary containing plist data
		"""
		# Use Foundation class to extract plist data (Foundation used instead of plistlib due to support for binary plists)
		plistNSData = Foundation.NSData.dataWithContentsOfFile_(plistFile)
		plistData, format, error = Foundation.NSPropertyListSerialization.propertyListFromData_mutabilityOption_format_errorDescription_(plistNSData, Foundation.NSPropertyListMutableContainersAndLeaves, None, None)
		
		return plistData
	
	def _parsePlists(self):
		""" Extract relevant plist fields from the internally built list of startup plists.  
		This function calls _readRawPlist to load plist data.
		
		@type raw_data: None
		@param row_data: None
		
		@return: dictionary containing startup items
		"""
		
		startupItems =[]
		for user, plistName in self.startup_plists:
			pEntry ={}
			pEntry['User']=user
			pEntry['Fname']=plistName
			plistData = self._readRawPlist(plistName)
			#print plistData
			
			try:
				# Begin Label String
				if plistData.has_key("Label")==True:
					pEntry["Label"]=plistData["Label"]
				# End Label String
			
				# Begin Program String
				progStr=""	
				if plistData.has_key("Program") == True:
					progStr=str(plistData["Program"])
						
				if plistData.has_key("ProgramArguments") == True:
					# If program exists then eliminate the arg of ProgramArguments if it matches basename(Program)
					for entry in plistData["ProgramArguments"]:
						if os.path.basename(progStr) != entry: # Remove first program arg if already specified by "Program"
							if len(progStr) > 0:
								progStr=progStr+" "

							progStr=progStr+str(entry)

				# This section is designed to support the alternate formate of the loginwindow.plist format.
				# This could have been handled more elegantly but would have forced a rewrite of several functions. 
				if plistName.endswith(LOGIN_PLIST):
					if plistData.has_key("AutoLaunchedApplicationDictionary") == True:
						for entry in plistData["AutoLaunchedApplicationDictionary"]:
							pEntry["Program"]=entry["Path"]
							pEntry["Disabled"]=False
							startupItems.append(pEntry.copy())

					pEntry=None

				else:
					if len(progStr) >0:
						pEntry["Program"]=progStr
					# End Program String
						
					# Begin Disabled Status
					if plistData.has_key("Disabled") == True:
						pEntry["Disabled"]=plistData["Disabled"]
					else:
						pEntry["Disabled"]=False
					# End Diabled Status	

			except:
				pEntry['Label'] = "ERROR"
				pEntry['Disabled'] = False
				pEntry['Program'] = "Error Parsing Plist(%s)" % (plistName)
			
			if pEntry != None:
				startupItems.append(pEntry)
			
		return startupItems
	
	# --- Section: Building List of Plists -------
	def _expandPlistDirs(self, mntPoint):
		""" Expand the plist entries to represent fullpaths to actual files.
		Startup plist configuration entries may contain a directory where plists can be found (/Library/LaunchDaemons) not full paths.
		This functions does the appropriate expansion to make the complete list.
		
		@type raw_data: string
		@param row_data: MointPoint (Default=/) of the OS X image to analyze
		
		@return: None, Updates internal startupitem list
		"""
		expandedPaths=[]

		# Convert Plist Directories into Full Path of All Plists
		for user, plistPath in self.startup_plists:
			# Do not prepend only /, would result in a path starting with //
			if mntPoint == "/":
				mntPoint=""
				
			plistPath=mntPoint+plistPath
		
			try:
				if os.path.isfile(plistPath) and plistPath.endswith(PLIST_EXT):
					expandedPaths.append([user, plistPath])
					
				elif os.path.isdir(plistPath):
					for root, subFolders, files in os.walk(plistPath):
						for file in files:
							if file.endswith(PLIST_EXT):
								fpath =os.path.join(root,file)
								expandedPaths.append([user, fpath])
		
			except IOError:
				print "Error reading: "+ plistPath
				sys.exit(1)
				
		self.startup_plists=expandedPaths
	
	def _expandUserPaths(self, mntPoint):
		""" Expand the plist entries to represent fullpaths to actual files.
		Startup plist configuration entries may use ~ short cuts to represent user directories.
		This functions does the appropriate substitutions and expansion to make the complete list.
		
		@type raw_data: string
		@param row_data: MointPoint (Default=/) of the OS X image to analyze
		
		@return: None, Updates internal startupitem list
		"""
		listOfPlists =[]
		
		if mntPoint == "/":
			mntPoint=""
		
		userDirs = self.user_list
		
		for plistPath in self.startup_plists:
			if plistPath.startswith("~") == True:
				for user,homedir in userDirs:
					listOfPlists.append([user, plistPath.replace("~", homedir)])
			else:
				listOfPlists.append(["SYSTEM", plistPath])
		
		self.startup_plists =listOfPlists	

	# ---- Section: UI & Formatting -------
	def _FormatedStartupItemsText(self, targetUser):
		""" Builds a text formatted list of startupitems for the specified user.
		
		@type raw_data: string
		@param row_data: Only show startup items related to specified user (Default is ALL)
		
		@return: List containing formatted output
		"""		
		rtnList =[]

		currentUser =""
		for cmd in self.getStartupItemsRaw(targetUser):
			if cmd.has_key('User') == True and cmd.has_key('Program') == True:
				if cmd['User'] != currentUser:
					rtnList.append(cmd['User'])
					currentUser = cmd['User']
					
				cmdStr="\t"+cmd['Program']
				if cmd['Disabled'] == True:
					cmdStr=cmdStr+"(Disabled)"
			else:
				cmdStr="\tError Reading Plist -- Fields 'User' and/or 'Program' does not exist\nEntry:" +str(cmd)
				
			rtnList.append(cmdStr)
			
		return rtnList
	
	def _FormatedStartupListCSV(self, targetUser):	
		""" Builds a csv formatted list of startupitems for the specified user.
		
		@type raw_data: string
		@param row_data: Only show startup items related to specified user (Default is ALL)
		
		@return: List containing formatted output
		"""	
		import csv
		import StringIO
		
		rtnList=StringIO.StringIO()
		startupItems = self.getStartupItemsRaw(targetUser)

		keys=startupItems[0].keys()
				
		csvWriter = csv.DictWriter(rtnList, keys, restval="NO VALUE", delimiter=',', skipinitialspace=True, quotechar='\"', lineterminator="", quoting=csv.QUOTE_ALL)

		csvWriter.writerow(dict((nHeader,nHeader) for nHeader in keys))
		
		for entry in startupItems:
			csvWriter.writerow(entry)
			
		return rtnList.buflist
	
	# ---- Section: Public Accessor Functions -----
	def getStartupPlists(self, targetUser="ALL"):
		""" Return a text list of startup plists for the specified user.
		This information is a subset of the information return from startupItems.
		
		@type raw_data: string
		@param row_data: Only show startup plists related to specified user (Default is ALL)
		
		@return: List of text entries
		"""	

		rtnList =[]
		
		for owner, plist in self.startup_plists:
			if targetUser=='ALL' or owner == 'SYSTEM' or owner == targetUser:
				rtnList.append(plist)
				
		return rtnList

	
	def getStartupCmds(self, user):
		""" Return a text list of startup commands for the specified user.
		This information is a subset of the information return from startupItems.
		
		@type raw_data: string
		@param row_data: Only show startup commands related to specified user (Default is ALL)
		
		@return: List of text entries
		"""	
		rtnList =[]
		
		for entry in self.getStartupItemsRaw(user):
			if entry.has_key('Program') == True:
				cmdStr=entry['Program']
				if entry.has_key('Disabled') == True and entry['Disabled'] == True:
					cmdStr=cmdStr+" (Disabled)"
			else:
				cmdStr=str("Error Reading Plist -- Field 'Program' does not exist")
				
			rtnList.append(cmdStr)
			
		return rtnList
	
	def getStartupItemsRaw(self, user="ALL"):
		""" Return a raw list of startupitems for the specified user.
		
		@type raw_data: string
		@param row_data: Only show startup items related to specified user (Default is ALL)
		
		@return: List containing dictionary entries
		"""	
		rtnList = []
		
		if user=="ALL":
			return self.startupItems
		else:
			for entry in self.startupItems:
				if entry.has_key('User') == True:
					if entry['User'] == 'SYSTEM' or entry['User'] == user:
						rtnList.append(entry)
				else:
					rtnList.append("Error Reading Plist -- Field 'User' does not exist")
					
		return rtnList
			
	def getStartupItemsText(self, targetUser="ALL"):
		""" Return a text list of startupitems by called _FormatedStartupXXXX functions.
		
		@type raw_data: string
		@param row_data: Only show startup items related to specified user (Default is ALL)
		
		@return: List containing text entries
		"""	
		itemList=self._FormatedStartupItemsText(targetUser)
		return itemList
	
	def getStartupItemsCSV(self, targetUser="ALL"):
		""" Return a csv list of startupitems by called _FormatedStartupXXXX functions.
		
		@type raw_data: string
		@param row_data: Only show startup items related to specified user (Default is ALL)
		
		@return: List containing text entries
		"""	
		itemList =self._FormatedStartupListCSV(targetUser)	
		return itemList
	
	# --- Section: Output Methods
	def outputStartupList (self, format, targetUser="ALL", filename=None):
		""" Output the list of startup items.  Formatting & Output destination specified.
		
		@type raw_data: string
		@param row_data: Format to be used in generating output
		@type raw_data: string
		@param row_data: Only show startup items related to specified user (Default is ALL)
		@type raw_data: string
		@param row_data: Output written to filename specified, None=stdout
		
		@return: None
		"""	
		if targetUser != "ALL" and targetUser != "System":
			if self._userExists(targetUser) == False:
				print "User: %s does not exist or incorrect permissions to access user information!\nPlease validate username and/or use elevated privileges" % (targetUser)
				sys.exit(1)
		
		if format == 'csv':
			itemList=self.getStartupItemsCSV(targetUser)
		elif format == 'raw':
			itemList=self.getStartupItemsRaw(targetUser)
		elif format == 'text':
			itemList=self.getStartupItemsText(targetUser)
		else:
			print "Invalid output format specified -o "+str(format)
			sys.exit(1)
			
		if filename == None:
			for entry in itemList:
				print entry
		else:
			try:
				fOut=open(filename, "w")
				for entry in itemList:
					fOut.write(str(entry)+"\n")
				fOut.close()
			except:
				print "Error: Unable to write output to file -- "+str(filename)
				sys.exit(1)
				
	def outputList (self, listType, targetUser="ALL", filename=None):
		""" Output the list of cmands or plists for a target user.  Formatting & Output destination specified.
		
		@type raw_data: string
		@param row_data: Format to be used in generating output
		@type raw_data: string
		@param row_data: Only show startup items related to specified user (Default is ALL)
		@type raw_data: string
		@param row_data: Output written to filename specified, None=stdout
		
		@return: None
		"""	
		if targetUser != "ALL" and targetUser != "System":
			if self._userExists(targetUser) == False:
				print "User: %s does not exist or incorrect permissions to access user information!\nPlease validate username and/or use elevated privileges" % (targetUser)
				sys.exit(1)
		
		if listType == 'plists':
			itemList =self.getStartupPlists(options.user)
			
		elif listType == 'cmds':
			itemList=self.getStartupCmds(options.user)
		else:
			print "Invalid List Type Specified: "+listType
			sys.exit(1)
			
		if filename == None:
			for entry in itemList:
				print entry
		else:
			try:
				fOut=open(filename, "w")
				for entry in itemList:
					fOut.write(entry+"\n")
				fOut.close()
			except:
				print "Error: Unable to write output to file -- "+str(filename)
				sys.exit(1)

if __name__ == '__main__':
	import optparse
	
	usage = "usage: %prog -o <output type> [options]"
	
	parser = optparse.OptionParser()
	parser.set_usage(usage)
	
	parser.add_option("-o", "--output", dest="output",
	                        help="Output format: csv, raw, text", metavar="<OUTPUT TYPE>", default=None)
	
	parser.add_option("-l", "--list", dest="list",
	                        help="List supporting items: cmds, plists", metavar="<LIST TYPE>", default=None)
		
	parser.add_option("-m", "--mountpoint", dest="mountpoint",
	                        help="Mac OS X Mount Point, Default = \"/\"", metavar="<MOUNT_POINT>", default="")

	parser.add_option("-f", "--file", dest="outfile",
	                        help="Send output to specified file instead of stdout", metavar="<FILE>", default=None)
	
	parser.add_option("-u", "--user", dest="user",
							help="Only show startup items for specified user (default = ALL)", metavar="<USERNAME>", default="ALL")
	
	parser.add_option("-V", "--verbose", action="store_true", dest="verbose",
	                        help="Verbose Output", default=False)
	
	(options, args) = parser.parse_args()        

	if options.mountpoint != "":
		if os.path.exists(options.mountpoint) == False:
			print "Error invalid mount point: " + str(options.mountpoint)
			sys.exit(1)

	if options.list != None:
		aruns=AutoRuns(options.verbose, options.user, options.mountpoint)
		aruns.outputList(options.list, options.user, options.outfile)

	elif options.output != None:
		aruns=AutoRuns(options.verbose, options.user, options.mountpoint)
		aruns.outputStartupList(options.output, options.user, options.outfile)

	else:
		parser.print_help()
		print ""
		print "Usage Error -- Must specify output type"
		sys.exit(1)

Categories: Apple, OS Internals

Does Windows have a Microkernel or Monolithic kernel?

February 20, 2011 4 comments

Like most Unix systems, Windows is a monolithic operating system. Why? Because the kernel mode protected memory space is shared by the operating system and device driver code.

But don’t take my word for it, from Microsoft’s Curriculum Resource Kit (written by Mark Russinovich and Dave Solomon, authors of Windows Internals):

Is Windows a microkernel-based OS?

  • No – not using the academic definition (OS components and drivers run in their own private address spaces, layered on a primitive microkernel)
  • All kernel components live in a common shared address space
  • Therefore no protection between OS and drivers

Why not pure microkernel?

  • Performance – separate address spaces would mean context switching to call basic OS services
  • Most other commercial OSs (Unix, Linux, VMS etc.) have the same design

But it does have some attributes of a microkernel OS

  • OS personalities running in user space as separate processes
  • Kernel-mode components don’t reach into one another’s data structures
  • Use formal interfaces to pass parameters and access and/or modify data structures

Therefore the term “modified microkernel”

Further reading at TechNet on the Windows NT 3.51 kernel reveals more relevant information:

A microkernel, on the other hand, is the name given to the core portion of a modern, modular operating system. Microkernel operating systems are based on two fundamental principles. The most basic principle is one of modularity, encapsulation, and data hiding. In this aspect of the design, there is one and only one portion of the operating system that has system-wide responsibility for a particular function.

All other parts of the operating system (as well as applications, naturally) access that function through a well-defined interface. There is no duplication of function and no “back doors” to critical data structures; all access to system data structures is through software interfaces. This approach makes it possible to upgrade or replace entire modules within the system without disturbing the remainder.

A secondary principal of the microkernel design, related to the first but focused more on the implementation strategy, is that large portions of the operating system which traditionally run entirely in the kernel or privileged mode of the microprocessor can now be executed in user or application mode, with only the microkernel itself, along with a relatively small amount of hardware device-specific code, executing in kernel mode.

Operating systems that follow both of these principles are often called pure microkernel systems. Operating systems that follow the first principle only strict modularity and strong encapsulation but not the second, are sometimes called modified microkernel or macrokernel operating systems.

Categories: OS Internals, Tech, Windows

The Elements of Computing Systems

June 17, 2010 Leave a comment

The Elements of Computing Systems: Building a Modern Computer from First Principles

This is a very interesting book, and course, that to me seems to take Charles Petzold’s book Code a step further by introducing free software that can be used to visualize fundamental computing concepts.

The description from Amazon sums it up quite succinctly:

In the early days of computer science, the interactions of hardware, software, compilers, and operating system were simple enough to allow students to see an overall picture of how computers worked. With the increasing complexity of computer technology and the resulting specialization of knowledge, such clarity is often lost. Unlike other texts that cover only one aspect of the field, The Elements of Computing Systems gives students an integrated and rigorous picture of applied computer science, as its comes to play in the construction of a simple yet powerful computer system.

Indeed, the best way to understand how computers work is to build one from scratch, and this textbook leads students through twelve chapters and projects that gradually build a basic hardware platform and a modern software hierarchy from the ground up. In the process, the students gain hands-on knowledge of hardware architecture, operating systems, programming languages, compilers, data structures, algorithms, and software engineering. Using this constructive approach, the book exposes a significant body of computer science knowledge and demonstrates how theoretical and applied techniques taught in other courses fit into the overall picture.

Designed to support one- or two-semester courses, the book is based on an abstraction-implementation paradigm; each chapter presents a key hardware or software abstraction, a proposed implementation that makes it concrete, and an actual project. The emerging computer system can be built by following the chapters, although this is only one option, since the projects are self-contained and can be done or skipped in any order. All the computer science knowledge necessary for completing the projects is embedded in the book, the only pre-requisite being a programming experience.

The book’s web site provides all tools and materials necessary to build all the hardware and software systems described in the text, including two hundred test programs for the twelve projects. The projects and systems can be modified to meet various teaching needs, and all the supplied software is open-source.

Categories: OS Internals, Programming, Tech

How to unpack a file using Process Explorer and WinDbg

June 14, 2010 4 comments

One of the big issues when analyzing malicious files is that DLL’s and EXE’s are often packed using executable ‘packing’ software. Sometimes this software is well known, other times it is could be a custom compression software which makes it even more difficult to unpack.

While it can be very tedious and difficult to statically analyze a binary that has been packed, by its very nature, when executed the file must decrypt itself in order to execute on the system.

Using the latest version of Process Explorer (v12.04) and WinDbg, I have figured out at least 1 way to dump a DLL or EXE from the system’s loaded memory to a file. This version of the DLL or EXE written back to disk is not the same as the packed original, but is in fact the unpacked version that has been loaded into the system’s memory.

This technique is useful because it allows an analyst to review an unpacked version of the malicious file. Enough background, let’s begin.

In this example, we will be attempting to unpack a potentially malicious DLL. Virus Total currently shows that 6/41 AV Vendors detect this as a Trojan known as Win32.Hiloti.

I copied the file, “ezoyenaxixib.dll” to my c:\temp folder on my sandboxed system. The first step is to launch Process Explorer.

The next step is to actually load this DLL on my sandbox system. Because I was able to capture the registry entry for this file, I know the command it intended to use in order to have the system launch it at logon:

rundll32.exe c:\temp\ezoyenaxixib.dll,startup

You may receive an error message that rundll32 could not find the entry point named “startup”. This is alright, but do NOT click OK. Doing so will cause rundll32.exe to close and to unload our malicious DLL. If you do mistakenly click OK, then just re-run the previous command.

clip01

Notice in the screenshot above that Process Explorer shows “rundll32.exe” as a child of cmd.exe, which we would expect since we launched it from a cmd prompt. Also notice that the command line shows rundll32.exe and our DLL.

Right-click on “rundll32.exe” in Process Explorer and select the Create Dump menu and select Create Full Dump …

image

 

For this example, we will save this file to c:\temp\rundll32_hiloti.dmp

Next, click again on “rundll32.exe” in order to highlight it. Now click on the View menu at the top of the Process Explorer Window and select the Lower Pane View menu and select DLLs:

image

(You also need to ensure that Show Lower Pane is checked). Notice that we now see our malicious DLL in the lower pane view. Go to the lower pane and right click on the malicious DLL and select Properties.

image

Once the properties dialog opens up, take note of 2 important pieces of information, the Load Address and the Mapped Size. Write these 2 values down if you need to.

image

NOTE that if you click on the Strings tab and select the Memory radio button, you will see the strings embedded in the unpacked version of this DLL that is loaded into memory. Sometimes this is all you may be looking for, and can quit here.

But, if you’d like to write this unpacked version of the DLL loaded in memory to a file on disk, continue on!

At this point, we can close Process Explorer and click OK on the Rundll32 error dialog box. Next you will need to launch WinDbg, which is part of the freely downloadable Windows Debugging Tools.

Once WinDbg is open, click the File menu and select Open Crash Dump. In this case, I am going to load the crash dump that I created with Process Explorer earlier, and saved to c:\temp\rundll32_hiloti.dmp.

image

Now, with our crash dump loaded by WinDbg, we can start entering commands (where the red arrow is pointing above). We are going to execute the .writemem command which allows us to write a section of memory to a file. Enter the following:

.writemem c:\temp\unpacked_hiloti.dll 0x10000000 L47000

So what does this all mean?

  • Well, c:\temp\unpacked_hiloti.dll is the name of the file we are creating and where this memory is written.
  • 0x10000000 is the Load Address of this DLL that Process Explorer gave us.
  • And finally, L47000 is the number of bytes to write from memory, which was given to us by Process Explorer. Note that WinDbg requires the prefix “L” when referring to an object value, such as the number of bytes

image

image

That’s it! At this point, you can now load the unpacked DLL into your favorite disassembler for further analysis.

Shell Policy is not the same as security

April 10, 2010 Leave a comment

Another classic post from Windows API developer, Raymond Chen. It’s an important reminder, and one that I find many people don’t fully understand.

Shell policies control how Explorer and other shell components behave, but that’s just blocking the front door.

For example, there is a shell policy to prevent the user from changing the wallpaper from the Desktop control panel. This disables the controls on the Desktop control panel for changing the wallpaper, but there are ways to change the wallpaper other than that. If users can run an arbitrary program, then they can run a program that calls SystemParametersInfo(SPI_SETDESKWALLPAPER) to change the wallpaper directly, bypassing the shell.

The purpose of the shell policies is merely to make it more difficult for users to perform various categories of operations by removing them from the shell interface. But, of course, if the users are allowed to write their own program with its own user interface, then they can still access the underlying functionality.

And this really sums it all up right here:

Setting a policy to remove the user interface for a feature is like removing the staircase that leads to the second floor to keep people out. If you let them bring a ladder, then they can still get up there.

And in a similar vein, Mark Russinovich (of PsTools and SysInternals fame), shows us how to bypass group policy as a limited user. Another reason I don’t think much of GPO’s…and I’m speaking as a former GPO administrator!

I’d also add (it may be obvious if you read Mark’s article) that if a users has Power User or Administrator credentials on their local system, then you have no guarantee that your group policies are securing anything. It’s trivial to override all GPO’s on your local system if you have these elevated rights.

Categories: OS Internals, Tech, Windows

Why does Task Manager show an application’s memory usage drop after minimizing it to the the taskbar?

January 8, 2010 4 comments

Have you ever tried this? Open up a program like Internet Explorer and open up a few tabs. Watch the memory usage in Task Manager. Now, minimize the application and watch the memory usage plummet. Even when you restore the application, the memory usage is still lower than it was before minimizing it.

So why does this happen? Is memory really being “freed”? In short, no, not at all…it is merely being paged out to disk, because when you minimize the application Windows makes an assumption that you are not actively using it and can move the memory from the limited (and fast) RAM to the larger (but slower) paging file on disk.

UPDATE: Barry corrected me a bit in the comments. It is not necessarily being paged out to disk, but rather Windows is trimming the application’s working set when it is minimized:

The application isn’t paged out to disk when it’s minimized. Rather, the working set is trimmed. The OS will page *in* working set pages for that application’s timeslices. But if physical memory pressure isn’t high, the OS isn’t going to suddenly start paging the application’s memory out to disk just because the app was minimized.

But don’t take my word for it, Delphi compiler engineer, Barry Kelly explains it well in response to a question on Stack Overflow.

Task Manager doesn’t show the total that the application has allocated from Windows. What it shows (by default) is the working set. The working set is a concept that’s designed to try and minimize page file thrashing in memory-constrained conditions. It’s basically all the pages in memory that the application touches on a regular basis, so to keep this application running with decent responsiveness, the OS will endeavour to keep the working set in physical memory.

On the theory that the user does not care much about the responsiveness of minimized applications, the OS trims their working set. This means that, under physical memory pressure, pages of virtual memory owned by that process are more likely to be paged out to disk (to the page file) to make room.

Most modern systems don’t have paging issues for most applications for most of the time. A severely page-thrashing machine can be almost indistinguishable from a crashed machine, with many seconds or even minutes elapsing before applications respond to user input.

So the behaviour that you are seeing is Windows trimming the working set on minimization, and then increasing it back up over time as the application, restored, touches more and more pages. It’s nothing like garbage collection.

If you’re interested in memory usage by an application under Windows, there is no single most important number, but rather a range of relevant numbers:

  • Virtual size – this is the total amount of address space reserved by the application. Address space (i.e. what pointers point to) may be unreserved, reserved, or committed. Unreserved memory may be allocated in the future, either by a memory manager, or by loading DLLs (the DLLs have to go somewhere in memory), etc.
  • Private working set – this is the pages that are private to this application (i.e. are not shared across multiple running applications, such that a change to one is seen by all), and are part of the working set (i.e. are touched frequently by the app).
  • Shareable working set – this is the pages in the working set that are shareable, but may or may not actually be shared. For example, DLLs or packages (BPLs) may be loaded into the application’s memory space. The code for these DLLs could potentially be shared across multiple processes, but if the DLL is loaded only once into a single application, then it is not actually shared. If the DLL is highly specific to this application, it is functionally equivalent to private working set.
  • Shared working set – this is the pages from the working set that are actually shared. One could image attributing the “cost” of these pages for any one application as the amount shared divided by the number of applications sharing the page.
  • Private bytes – this is the pages from the virtual address space which are committed by this application, and that aren’t shared (or shareable) between applications. Pretty much every memory allocation by an application’s memory manager ends up in this pool. Only pages that get used with some frequency need become part of the working set, so this number is usually larger than the private working set. A steadily increasing private bytes count indicates either a memory leak or a long-running algorithm with large space requirements.

These numbers don’t represent disjoint sets. They are different ways of summarizing the states of different kinds of pages. For example, working set = private working set + shareable working set.

Which one of these numbers is most important depends on what you are constrained by. If you were trying to do I/O using memory mapped files, the virtual size will limit how much memory you can devote to the mapping. If you are in a physical-memory constrained environment, you want to minimize the working set. If you have many different instances of your application running simultaneously, you want to minimize private bytes and maximize shared bytes. If you are producing a bunch of different DLLs and BPLs, you want to be sure that they are actually shared, by making sure their load addresses don’t cause them to clash and prevent sharing.

About SetProcessWorkingSetSize:

Windows usually handles the working set automatically, depending on memory pressure. The working set does not determine whether or not you’re going to hit an out of memory (OOM) error. The working set used to make decisions about paging, i.e. what to keep in memory and what to leave on disk (in the case of DLLs) or page out to disk (other committed memory). It won’t have any effect unless there is more virtual memory allocated than physical memory in the system.

As to its effects: if the lower bound is set high, it means the process will be hostile to other applications, and try to hog memory, in situations of physical memory pressure. This is one of the reasons why it requires a security right, PROCESS_SET_QUOTA.

If the upper bound is set low, it means that Windows won’t try hard to keep pages in physical memory for this application, and that Windows may page most of it out to disk when physical memory pressure gets high.

In most situations, you don’t want to change the working set details. Usually it’s best to let the OS handle it. It won’t prevent OOM situations. Those are usually caused by address space exhaustion, because the memory manager couldn’t commit any more memory; or in systems with insufficient page file space to back committed virtual memory, when space in the page file runs out.

Categories: OS Internals, Tech, Windows

Investigating Prefetch Files

January 8, 2010 Leave a comment

Another awesome utility from Nils Sofer at Nirsoft. This application allows you to dig into prefetch files that are created on your system to aid in speeding up the cold boot time of an application. The following is taken directly from Nirsoft’s blog.

Each time that you run an application in your system, a Prefetch file (.pf file) which contains information about the files loaded by the application, is created by Windows operating system. The information in the Prefetch file is used for optimizing the loading time of the application in the next time that you run it. These Prefetch files are stored in C:\Windows\Prefetch, starting from Windows XP.
WinPrefetchView is a new utility that allows you to easily watch the content of these .pf files.  By looking in these files, you can learn which files every application is using, and which files are loaded on Windows boot.

For more information about this new utility, click here.

 winprefetchview

Categories: OS Internals, Tech, Windows