VMware Transparent Page Sharing vs Ubuntu Address Space Layout Randomization (I)

Passed VCP5 few weeks ago, and while studying (hardly, it’s not an easy one!), I suddenly spotted two opposed concepts: VMware TPS (Transparent Page Sharing), and ASLR (Address Space Layout Randomization).

I’ve first heard about ASLR while playing with OpenBSD, and reading about their security features (they go far more than just randomizing memory pages!). When studying for the VCP5 exam, I realized that shared pages should suffer when memory pages get randomized: starting two identical machines could end with few identical memory pages to be shared.

After the exam, I started to look for performance tests about shared pages when ASLR is active, and found a very detailed blog entry called Windows 7 Transparent Page Sharing and the ASLR story. They found that ASLR could reduce TPS effectivity, but not too much.

I couldn’t resist the temptation of doing my own tests. My test lab is a cheap ESX5 HP ProLiant ML110 G5 server with 8Gb RAM, and a W2K3 vCenter virtual machine running in the PowerBook that I’m writing now.

I created an Ubuntu 10.10 32bit virtual machine with 1Gb of memory, installed VMware tools, and clonned it 15 times. Ubuntu comes with ASLR activated by default, so when all the virtual machines powered on, I thought that TPS will not do a good job: I was wrong, 1.85Gb of memory where shared after a few hours. Not too bad.

Then I turned off ASLR in every Ubuntu machine, running “sudo sysctl -w kernel.randomize_va_space=0”, and restarted all the virtual machines. After a few hours, TPS found 2.83Gb of shared memory.

Much better, but if all the machines are basically the same one and ASLR is turned off, why there are only 2.83Gb shared from the 15Gb of all the assigned memory?.

The response is probably that Ubuntu machines have 1Gb assigned, but they are doing nothing, there is no memory pressure over the ESX: memory is overcommitted, but in the actual circumstances there is no swapping, no balloon and plenty of unaccessed memory.

But what will happen if I power on twenty 1Gb machines in my 8Gb ESX and wait for a few hours?…

This is when VMware shines, when all the memory reclamation techniques start to work: you can see balloon, compressed and zero reclaiming adding to TPS, and that only 249Mb have been paged to disk.

When you look to each VM, you can see that machines with low Shared and Unaccessed memory values, have started to Balloon memory.

VM	Consumed	Shared	Ballooned	Unaccessed
ubu00 ubu01 ubu02 ubu03 ubu04 ubu05 ubu06 ubu07 ubu08 ubu09 ubu10 ubu11 ubu12 ubu13 ubu14 ubu15 ubu16 ubu17 ubu18 ubu19	284 134 123 336 131 214 414 248 399 407 129 165 206 272 117 133 336 355 252 127	789 236 211 129 218 861 239 821 289 463 229 901 849 728 212 217 734 303 390 237	0 649 649 0 649 0 0 0 0 0 649 0 0 0 649 649 0 0 0 649	23 40 56 602 56 3 426 16 392 212 39 11 21 69 53 34 11 425 430 34

But, why some machines had high Unaccessed memory values and other have “accessed” all their memory?. It seems that linux uses all the free memory as a disk cache (see Linux ate my RAM), so I restarted all the Virtual Machines slowly, waiting for the down to 0Mhz of each Ubuntu vCPU before starting another one, and… all the Virtual Machines started, without Ballooning, or swapping!!!

Well, the trick is that every VM is doing nothing, and full of unaccessed memory. Moreover, remember that ASLR is turned off, and while waiting for the vCPU the TPS started it’s job, and the ESX was never out of free host memory.

VM	Consumed	Shared	Ballooned	Unaccessed
ubu00 ubu01 ubu02 ubu03 ubu04 ubu05 ubu06 ubu07 ubu08 ubu09 ubu10 ubu11 ubu12 ubu13 ubu14 ubu15 ubu16 ubu17 ubu18 ubu19	286 288 350 287 283 285 264 282 280 283 308 362 286 365 292 232 177 162 166 163	66 64 78 67 65 67 263 67 68 66 75 68 63 63 151 125 189 209 205 208	0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0	713 713 643 713 718 713 540 717 717 717 641 637 715 637 647 720 722 719 724 723

I left these machines powered on for one day, and the same pattern emerged: some machines ballooning, others plenty of unaccessed memory, and the rest with high shared values.

VM	Consumed	Shared	Ballooned	Unaccessed
ubu00 ubu01 ubu02 ubu03 ubu04 ubu05 ubu06 ubu07 ubu08 ubu09 ubu10 ubu11 ubu12 ubu13 ubu14 ubu15 ubu16 ubu17 ubu18 ubu19	303 290 251 158 279 295 189 322 305 300 245 316 304 263 156 167 134 307 271 406	590 248 835 231 800 513 884 383 162 173 842 724 160 822 270 896 229 608 816 257	0 0 0 649 0 0 0 0 0 0 0 0 0 0 623 0 649 0 0 0	179 529 11 33 4 257 11 360 599 595 5 27 604 18 22 19 49 151 5 427

You can see how part of the unaccessed memory has been used by Ubuntu, giving ESXi the chance to find more shared pages, ending with an incredible 10.3Gb of shared memory!.

Previous tests were done with ASLR turned off, so I logged into each Ubuntu, and run “sudo sysctl -w kernel.randomize_va_space=2”. Then restarted all the virtual machines, and waited for another full day…

You can see 9.2 Gb of shared memory with ASLR activated!. Think about it: my host has only 8Gb, and is running the ESXi plus 20 virtual machines with 1Gb assigned to each one.

It’s true that these virtual machines are only running system processes. But do you remember those memory oversized virtual servers that your boss wanted: after a few hours running they will end with a few hundred Mb of active memory, and TPS will recover all those shared pages (even with ASLR activated 🙂

In the next post, I will make some tests using “lookbusy” for generating memory load over two virtual machines, and testing again how ASLR affect TPS in a more real situation.

 

 

Notify 0.1.0, simply notification for windows

This is a simple command line tool, that shows a notification and can react if user clicks, closes or waits until timeout.

Use:

notify.exe “title” “text” “timeout” “clickcommand” “closecommand” “timeoutcommand” “minwidth” “maxwidth”

• title – Title of your notification
• text- Text of your notification
• timeout [60]- timeout in seconds
• clickcommand [“”]- launched when clicked
• closecommand [“”]- launched when closed
• timeoutcommand [“”]- launched when timeout
• minwidth [200]- minimum notification width in pixels
• maxwidth [600]- maximum notification width in pixels

Only ‘title’ and ‘text’ are required.

You should use double quotation marks for each parameter.

You can use ‘n’ to insert a new line inside ‘text’.

Width and height are calculated automatically.

If any word is longer than maxwidth, notify returns with error 1 (be careful with URLs).

Every command is lauched with hidden window.

If you want to show the window, launch command with ‘cmd /c “”command””‘.

The icon can be changed modifing ‘notify.jpg’.See the examples for use cases.

Thanks to Melba23 for StringSize and to Jonathan Bennett and the AutoIt Team.

Download from: https://github.com/ferfebles/Notify

Spray: new functionality

I’ve added new functionality to Spray, cleaned some code, resolved several bugs, and uploaded a new video.

• Ruby debugger integrated in a REPL tab (variables, eval, conditional breakpoints… and much more)
• ToolBar debug buttons: debug, breakpoint, next, step, continue and list.
• Tags current line in editor tab.
• Breakpoint persistence between restarts. 

You can download it from github: https://github.com/ferfebles/spray

It has been sparsely tested with Redcar 0.11 and current, over OSX, Linux and Windows XP.

And now, the improved video! (download for better quality).

Spray: basic, simple (beta) debugger plugin for redcar editor

Since Netbeans dropped Ruby support, I was looking for a ruby IDE with support for Linux, Windows and OSX.

After reviewing some options, started to play with Redcar editor. It’s clean, fast, simple, and written in Ruby. You can write your own plugins, and believe me, it’s really easy.

Moving from Netbeans, I missed some things: over all, the debugger. So I started to play with rdebug command line, and thought: “could rdebug be integrated in redcar?”.

In two weeks of testing, hacking, and reading Redcar and rdebug documentation, I wrote the Spray (beta) plugin.

It lacks:

• Variables, eval, and call stack information.
• Conditional breakpoints.

It has:

• Ruby debugger integrated in a REPL tab (variables, eval, conditional breakpoints… and much more).
• ToolBar debug buttons: debug, next, step, continue and list.
• Tags current line in editor tab.

Working in:

• Breakpoint toolbar button.
• Breakpoint persistence between restarts.

 

If you use rdebug, probably this is more than you ever needed.

If you use Netbeans or Rubymine, this could seem like a wombat against a sabretooth.

If you use Redcar, take a look at http://blog.bithug.org/2011/04/redcar-debug, it’s far more advanced than this project.

Why would you like to test this plugin?

 

• It gives access to the rdebug command line power: conditional breakpoints, frames…
• It makes rdebug easier to use.
• It’s small and hackable. You can add your own commands or support to new languages.

If you still want to install this:

• Install ruby-debug gem.
• Download and copy the spray folder to your redcar plugins folder.
• Restart redcar, and you can see the new buttons in the redcar ToolBar.

SprayTest.swf

VMWare Fusion virtual switch

Your custom internal network, shared between virtual machines.

VMWare ESX or ESXi allows for complex networking topologies: you can create virtual switches, and connect your machines with several virtual nics, to any of your virtual switches. But with VMWare Fusion you are limited to NAT, bridged or Host-only network modes.

There is a fourth choice: custom networking. This allows you to create a lab test network, where your virtual machines are interconnected. One of them, using another bridged nic, could act as DHCP, DNS and router.

You need your virtual machines working, powered off, and VMWare Fusion closed.

Open Terminal, go to the Fusion application folder:

cd /Library/Application Support/VMware Fusion/

Then, edit the file called networking. There are two virtual nics: VNET_1 y VNET_8. Add VNET_2: this will be the nic that virtual machines use for the internal network.

My file looks like this:

VERSION=1,0
answer VNET_1_DHCP yes
answer VNET_1_DHCP_CFG_HASH 238727CFAEBC008D3599C3EA810CC7B52B19E258
answer VNET_1_HOSTONLY_NETMASK 255.255.255.0
answer VNET_1_HOSTONLY_SUBNET 192.168.185.0
answer VNET_1_VIRTUAL_ADAPTER yes
answer VNET_2_DHCP no
answer VNET_2_VIRTUAL_ADAPTER yes
answer VNET_8_DHCP yes
answer VNET_8_DHCP_CFG_HASH 58D623D690468CFC4C11806A21CA8864FF5BC710
answer VNET_8_HOSTONLY_NETMASK 255.255.255.0
answer VNET_8_HOSTONLY_SUBNET 172.16.170.0
answer VNET_8_NAT yes
answer VNET_8_VIRTUAL_ADAPTER yes

Copy the two lines for VNET_2 to your network file.

Now, go to your virtual machines folder, select the virtual machine you want to connect to VNET_2, and select show package content. 

Edit your VirtualMachine.vmx file and change these ethernet0 options

ethernet0.connectionType = “custom”
ethernet0.vnet=”VMnet2″
ethernet0.linkStatePropagation.enable = “FALSE”

The third option makes that the ethernet0 of your Virtual Machine remains active, even when in your Mac, the VNET_2 virtual adapter is not initialized.

Change the VirtualMachine.vmx of the other virtual machines that you want to connect to your internal network.

Take care of not to edit your network setting from VMWare Fusion, as it would overwrite your custom network configuration.

Then, restart the VMWare Fusion networking:

sudo /Library/Application Support/VMware Fusion/boot.sh –restart  

Your virtual machines should now be connected to your own internal network.

 

This configuration is working in my computer with Fusion 3.1.2 over OSX 10.6.6. But this information is provided as is, without warranties, and you are advised to make at least 3 different backups of your hard disks and fridge notes. If you find any mistake or inaccuracy, please leave a comment.

More (probably better) information can be found here:

• VMware Fusion Virtual Network Configuration
• Adding VMNet’s in VMWare Fusion 3
• Network settings: VMX-file parameters