Crow Food

U.S.S Enterprise

2010-06-25T06:19:00.001-07:00

So I have been talking to other admins and one thing everyone brags/talks about strikes me as wrong.

Admin #1: I have 400 machines doing 600 mb/sec!
Admin #2: I have 60 machines doing 450 mb/sec.
Admin #1: Pfft only 60? I obviously know more then you, I have more servers, and you can’t understand the “enterprise” environment.

Now obviously that is not an exact translation of the conversation, but the gist of it. The part that really bugs me is 9 out of 10 admins will miss the fact admin #2 is doing much more traffic per server then admin #1. That means not only is the physical footprint smaller, but the environmental costs (power, cooling, etc) are lower as well. Bigger is not always (or almost ever) better. People should respect admin #2 for a more streamlined infrastructure and tuning his/her machines better.

That is not to say there can’t be a good reason for a large number of servers. The nature of the traffic/work can dictate a large number of spare machines to handle spikes in the load. However, a large number of sites have a vast number of machines just due to the laziness of the admins, and not due to any real business requirement. But hey, it’s Enterprise!

I don’t want to get started on some general enterprise rant, but I will say it seems to be an excuse to spend a lot of money on a lot of unneeded stuff, so that people wont have to do any more then contact some vendor and have them do the work. These systems generally don’t run any better then if some monkey typed out “yum install TheWorld” and ran ubuntu.

NFS speed tests with FileBench

2010-06-25T05:56:00.000-07:00

So I decided to do some more tests. These just are mucking around with different server and mount settings. In the end NFSv3 with noatime was the winner, but they are all very close.

Dell R610 with a PowerVault MD1120

R610:
12 GB ram
2x L5520 2.26Ghz 8m cache
2x 73GB 15k sas drives
Perc 6/iR controller

Dell MD1120
16x 126gb 15k rpm sas drives
2x 73 GB 15k sas drives

test1:
mount options: rsize=32768,wsize=32768,intr,vers=3,timeo=14,xattr

22320: 356.885: Per-Operation Breakdown
appendlog 3136ops/s 24.5mb/s 10.4ms/op 82us/op-cpu
closefile10 3135ops/s 0.0mb/s 0.0ms/op 15us/op-cpu
readfile10 3135ops/s 49.0mb/s 0.1ms/op 40us/op-cpu
openfile10 3135ops/s 0.0mb/s 1.2ms/op 88us/op-cpu
closefile9 3135ops/s 0.0mb/s 0.0ms/op 15us/op-cpu
readfile9 3135ops/s 49.0mb/s 0.1ms/op 40us/op-cpu
openfile9 3135ops/s 0.0mb/s 1.2ms/op 88us/op-cpu
closefile8 3135ops/s 0.0mb/s 0.0ms/op 15us/op-cpu
readfile8 3135ops/s 49.0mb/s 0.1ms/op 40us/op-cpu
openfile8 3135ops/s 0.0mb/s 1.2ms/op 87us/op-cpu
closefile7 3135ops/s 0.0mb/s 0.0ms/op 15us/op-cpu
readfile7 3135ops/s 49.0mb/s 0.1ms/op 40us/op-cpu
openfile7 3135ops/s 0.0mb/s 1.2ms/op 88us/op-cpu
closefile6 3135ops/s 0.0mb/s 0.0ms/op 15us/op-cpu
readfile6 3135ops/s 49.0mb/s 0.1ms/op 40us/op-cpu
openfile6 3135ops/s 0.0mb/s 1.3ms/op 88us/op-cpu
closefile5 3135ops/s 0.0mb/s 0.0ms/op 15us/op-cpu
readfile5 3135ops/s 48.9mb/s 0.1ms/op 40us/op-cpu
openfile5 3135ops/s 0.0mb/s 1.3ms/op 88us/op-cpu
closefile4 3136ops/s 0.0mb/s 0.0ms/op 15us/op-cpu
readfile4 3136ops/s 49.1mb/s 0.1ms/op 40us/op-cpu
openfile4 3136ops/s 0.0mb/s 1.3ms/op 88us/op-cpu
closefile3 3136ops/s 0.0mb/s 0.0ms/op 15us/op-cpu
readfile3 3136ops/s 48.9mb/s 0.1ms/op 40us/op-cpu
openfile3 3136ops/s 0.0mb/s 1.3ms/op 89us/op-cpu
closefile2 3136ops/s 0.0mb/s 0.0ms/op 15us/op-cpu
readfile2 3136ops/s 49.0mb/s 0.1ms/op 40us/op-cpu
openfile2 3136ops/s 0.0mb/s 1.4ms/op 89us/op-cpu
closefile1 3136ops/s 0.0mb/s 0.0ms/op 15us/op-cpu
readfile1 3136ops/s 49.0mb/s 0.1ms/op 40us/op-cpu
openfile1 3136ops/s 0.0mb/s 1.6ms/op 92us/op-cpu

22320: 356.886:
IO Summary: 29451150 ops, 97199.6 ops/s, (31355/3136 r/w) 514.6mb/s, 153us cpu/op, 2.3ms latency
22320: 356.886: Shutting down processes

test2:

mount options the same as #1, with added zfs set atime=off

22334: 346.293: Per-Operation Breakdown
appendlog 3183ops/s 24.9mb/s 9.5ms/op 79us/op-cpu
closefile10 3182ops/s 0.0mb/s 0.0ms/op 15us/op-cpu
readfile10 3182ops/s 49.7mb/s 0.1ms/op 40us/op-cpu
openfile10 3182ops/s 0.0mb/s 1.2ms/op 90us/op-cpu
closefile9 3182ops/s 0.0mb/s 0.0ms/op 15us/op-cpu
readfile9 3182ops/s 49.7mb/s 0.1ms/op 40us/op-cpu
openfile9 3183ops/s 0.0mb/s 1.2ms/op 90us/op-cpu
closefile8 3183ops/s 0.0mb/s 0.0ms/op 15us/op-cpu
readfile8 3183ops/s 49.7mb/s 0.1ms/op 40us/op-cpu
openfile8 3183ops/s 0.0mb/s 1.2ms/op 90us/op-cpu
closefile7 3183ops/s 0.0mb/s 0.0ms/op 15us/op-cpu
readfile7 3183ops/s 49.7mb/s 0.1ms/op 40us/op-cpu
openfile7 3183ops/s 0.0mb/s 1.3ms/op 90us/op-cpu
closefile6 3183ops/s 0.0mb/s 0.0ms/op 15us/op-cpu
readfile6 3183ops/s 49.7mb/s 0.1ms/op 40us/op-cpu
openfile6 3183ops/s 0.0mb/s 1.3ms/op 90us/op-cpu
closefile5 3183ops/s 0.0mb/s 0.0ms/op 15us/op-cpu
readfile5 3183ops/s 49.7mb/s 0.1ms/op 40us/op-cpu
openfile5 3183ops/s 0.0mb/s 1.3ms/op 90us/op-cpu
closefile4 3183ops/s 0.0mb/s 0.0ms/op 15us/op-cpu
readfile4 3183ops/s 49.7mb/s 0.1ms/op 40us/op-cpu
openfile4 3183ops/s 0.0mb/s 1.3ms/op 90us/op-cpu
closefile3 3183ops/s 0.0mb/s 0.0ms/op 15us/op-cpu
readfile3 3183ops/s 49.8mb/s 0.1ms/op 40us/op-cpu
openfile3 3183ops/s 0.0mb/s 1.3ms/op 91us/op-cpu
closefile2 3183ops/s 0.0mb/s 0.0ms/op 15us/op-cpu
readfile2 3183ops/s 49.7mb/s 0.1ms/op 40us/op-cpu
openfile2 3183ops/s 0.0mb/s 1.4ms/op 91us/op-cpu
closefile1 3183ops/s 0.0mb/s 0.0ms/op 15us/op-cpu
readfile1 3183ops/s 49.8mb/s 0.1ms/op 40us/op-cpu
openfile1 3183ops/s 0.0mb/s 1.6ms/op 94us/op-cpu

22334: 346.293:
IO Summary: 29894014 ops, 98660.8 ops/s, (31826/3183 r/w) 522.3mb/s, 155us cpu/op, 2.2ms latency
22334: 346.293: Shutting down processes

test3:

All the same options as test#2, however NFS switched to version 4.

22349: 348.183: Per-Operation Breakdown
appendlog 3137ops/s 24.5mb/s 0.5ms/op 46us/op-cpu
closefile10 3137ops/s 0.0mb/s 1.3ms/op 65us/op-cpu
readfile10 3137ops/s 49.0mb/s 0.1ms/op 37us/op-cpu
openfile10 3137ops/s 0.0mb/s 1.4ms/op 105us/op-cpu
closefile9 3137ops/s 0.0mb/s 1.3ms/op 65us/op-cpu
readfile9 3137ops/s 49.0mb/s 0.1ms/op 37us/op-cpu
openfile9 3137ops/s 0.0mb/s 1.5ms/op 106us/op-cpu
closefile8 3137ops/s 0.0mb/s 1.3ms/op 65us/op-cpu
readfile8 3137ops/s 49.0mb/s 0.1ms/op 37us/op-cpu
openfile8 3137ops/s 0.0mb/s 1.4ms/op 105us/op-cpu
closefile7 3137ops/s 0.0mb/s 1.3ms/op 65us/op-cpu
readfile7 3137ops/s 49.0mb/s 0.1ms/op 37us/op-cpu
openfile7 3137ops/s 0.0mb/s 1.5ms/op 106us/op-cpu
closefile6 3137ops/s 0.0mb/s 1.4ms/op 65us/op-cpu
readfile6 3137ops/s 49.0mb/s 0.1ms/op 37us/op-cpu
openfile6 3137ops/s 0.0mb/s 1.4ms/op 106us/op-cpu
closefile5 3137ops/s 0.0mb/s 1.4ms/op 65us/op-cpu
readfile5 3137ops/s 49.1mb/s 0.1ms/op 37us/op-cpu
openfile5 3137ops/s 0.0mb/s 1.5ms/op 106us/op-cpu
closefile4 3137ops/s 0.0mb/s 1.4ms/op 65us/op-cpu
readfile4 3137ops/s 49.0mb/s 0.1ms/op 37us/op-cpu
openfile4 3137ops/s 0.0mb/s 1.5ms/op 106us/op-cpu
closefile3 3137ops/s 0.0mb/s 1.4ms/op 65us/op-cpu
readfile3 3137ops/s 49.0mb/s 0.1ms/op 37us/op-cpu
openfile3 3137ops/s 0.0mb/s 1.5ms/op 106us/op-cpu
closefile2 3137ops/s 0.0mb/s 1.4ms/op 65us/op-cpu
readfile2 3137ops/s 49.0mb/s 0.1ms/op 37us/op-cpu
openfile2 3137ops/s 0.0mb/s 1.5ms/op 106us/op-cpu
closefile1 3137ops/s 0.0mb/s 1.4ms/op 66us/op-cpu
readfile1 3137ops/s 49.1mb/s 0.1ms/op 38us/op-cpu
openfile1 3137ops/s 0.0mb/s 1.6ms/op 112us/op-cpu

22349: 348.183:
IO Summary: 29467590 ops, 97253.8 ops/s, (31372/3137 r/w) 514.9mb/s, 207us cpu/op, 2.7ms latency
22349: 348.183: Shutting down processes

Test #4

All the options of test #2, however server set to 1024 instances.

22353: 413.961: Per-Operation Breakdown
appendlog 3142ops/s 24.5mb/s 0.4ms/op 46us/op-cpu
closefile10 3141ops/s 0.0mb/s 1.3ms/op 65us/op-cpu
readfile10 3141ops/s 49.1mb/s 0.1ms/op 37us/op-cpu
openfile10 3141ops/s 0.0mb/s 1.4ms/op 106us/op-cpu
closefile9 3141ops/s 0.0mb/s 1.3ms/op 65us/op-cpu
readfile9 3141ops/s 49.1mb/s 0.1ms/op 37us/op-cpu
openfile9 3141ops/s 0.0mb/s 1.4ms/op 106us/op-cpu
closefile8 3141ops/s 0.0mb/s 1.3ms/op 65us/op-cpu
readfile8 3141ops/s 49.1mb/s 0.1ms/op 37us/op-cpu
openfile8 3141ops/s 0.0mb/s 1.5ms/op 106us/op-cpu
closefile7 3141ops/s 0.0mb/s 1.3ms/op 65us/op-cpu
readfile7 3141ops/s 49.1mb/s 0.1ms/op 37us/op-cpu
openfile7 3142ops/s 0.0mb/s 1.5ms/op 106us/op-cpu
closefile6 3142ops/s 0.0mb/s 1.4ms/op 65us/op-cpu
readfile6 3142ops/s 49.1mb/s 0.1ms/op 37us/op-cpu
openfile6 3142ops/s 0.0mb/s 1.5ms/op 106us/op-cpu
closefile5 3142ops/s 0.0mb/s 1.4ms/op 65us/op-cpu
readfile5 3142ops/s 49.1mb/s 0.1ms/op 37us/op-cpu
openfile5 3142ops/s 0.0mb/s 1.5ms/op 106us/op-cpu
closefile4 3142ops/s 0.0mb/s 1.4ms/op 65us/op-cpu
readfile4 3142ops/s 49.1mb/s 0.1ms/op 37us/op-cpu
openfile4 3142ops/s 0.0mb/s 1.5ms/op 106us/op-cpu
closefile3 3142ops/s 0.0mb/s 1.4ms/op 65us/op-cpu
readfile3 3142ops/s 49.1mb/s 0.1ms/op 37us/op-cpu
openfile3 3142ops/s 0.0mb/s 1.5ms/op 107us/op-cpu
closefile2 3142ops/s 0.0mb/s 1.4ms/op 66us/op-cpu
readfile2 3142ops/s 49.1mb/s 0.1ms/op 37us/op-cpu
openfile2 3142ops/s 0.0mb/s 1.5ms/op 107us/op-cpu
closefile1 3142ops/s 0.0mb/s 1.4ms/op 66us/op-cpu
readfile1 3142ops/s 49.1mb/s 0.1ms/op 37us/op-cpu
openfile1 3142ops/s 0.0mb/s 1.6ms/op 112us/op-cpu

22353: 413.961:
IO Summary: 29508226 ops, 97387.8 ops/s, (31415/3142 r/w) 515.6mb/s, 208us cpu/op, 2.7ms latency
22353: 413.961: Shutting down processes

2010-02-02T13:45:00.000-08:00

Taking a look at OpenSolaris Indiana now that Nevada is dead (R.I.P.).

The first thing I see is massive bloat. Going to take awhile to cut this thing down to a reasonable production install size. JeOS looks promising, will have to watch that one more.

I did a default CD install, came out to over 12gb. That is a bit absurd. Looking closer at the installed packages there are about 650 installed., of which under 40 are in "Core". Seems a good place to start.

oddly enough sendmail, ppp and p7zip are now considered core. Here is a list I'm starting to build

Name: SUNWsprot
Summary: Solaris Bundled tools
Category: System/Core

Name: SUNWmd
Summary: Solaris Volume Manager
Category: System/Core

Name: SUNWcar
Summary: Core Architecture
Category: System/Core

Name: SUNWcs
Summary: Core Solaris
Category: System/Core

Name: SUNWcpc
Summary: CPU Performance Counter driver
Category: System/Core

Name: SUNWsolnm
Summary: Solaris Naming Enabler
Category: System/Core

Name: SUNWbtool
Summary: CCS tools bundled with SunOS
Category: System/Core

Name: SUNWrcmdc
Summary: Remote Network Client Commands
Category: System/Core

Name: SUNWpool
Summary: Resource Pools
Category: System/Core

Name: SUNWtcpd
Summary: tcpd - access control facility for internet services
Category: System/Core

Name: SUNWos86r
Summary: Platform Support, OS Functionality
Category: System/Core

Name: SUNWftp
Summary: FTP Server
Category: System/Core

Name: SUNWcnetr
Summary: Core Solaris Network Infrastructure
Category: System/Core

Name: SUNWfss
Summary: Fair Share Scheduler
Category: System/Core

Name: SUNWcsd
Summary: Core Solaris Devices
Category: System/Core

Name: SUNWcsl
Summary: Core Solaris, (Shared Libs)
Category: System/Core

Name: SUNWkvm
Summary: Core Architecture, (Kvm)
Category: System/Core

Name: SUNWmda
Summary: Solaris Volume Manager Assistant
Category: System/Core

Name: SUNWfwflash
Summary: Sun Firmware Flash Update Tool (fwflash)
Category: System/Core

Name: SUNWpoold
Summary: Dynamic Resource Pools
Category: System/Core

Name: SUNWcpcu
Summary: CPU Performance Counter libraries and utilities
Category: System/Core

Name: SUNWrcmds
Summary: Remote Network Server Commands
Category: System/Core

Name: SUNWzone
Summary: Solaris Zones
Category: System/Core

Name: SUNWgrub
Summary: GNU GRUB - GNU GRand Unified Bootloader
Category: System/Core

Name: SUNWckr
Summary: Core Solaris Kernel
Category: System/Core

Name: SUNWcakr
Summary: Core Solaris Kernel Architecture
Category: System/Core

Name: SUNWpicl
Summary: PICL Libraries, and Plugin Modules
Category: System/Core

Name: SUNWstosreg
Summary: OpenSolaris Registration
Category: System/Core

Name: SUNWrmodr
Summary: Realmode Modules
Category: System/Core

Name: SUNWrmodu
Summary: Realmode Modules
Category: System/Core

Name: SUNWsndm
Summary: Sendmail
Category: System/Core

Name: SUNWpppdt
Summary: Solaris PPP Tunneling
Category: System/Core

Name: SUNWbeadm
Summary: beadm application
Category: System/Core

Name: SUNWrsgk
Summary: kernel RPCSEC_GSS
Category: System/Core

Name: SUNWwbsup
Summary: WAN boot support
Category: System/Core

Name: SUNWipc
Summary: Interprocess Communications
Category: System/Core

Name: SUNWroute
Summary: Network Routing daemons/commands
Category: System/Core

Name: SUNWdistro-license-copyright
Summary: OpenSolaris licensing
Category: System/Core

Name: SUNWesu
Summary: Extended System Utilities
Category: System/Core

Name: SUNWp7zip
Summary: The p7zip compression and archiving utility
Category: System/Core

Somedays

2010-01-21T05:33:00.000-08:00

The little things make me happy.

[root@webnfs04:]% format
Searching for disks...done

AVAILABLE DISK SELECTIONS:
0. c0t0d0
/pci@0,0/pci8086,2940@1c/pci1028,1f10@0/sd@0,0
1. c3t0d0
/pci@0,0/pci8086,340e@7/pci1028,1f0a@0/sd@0,0
2. c3t1d0
/pci@0,0/pci8086,340e@7/pci1028,1f0a@0/sd@1,0
3. c3t2d0
/pci@0,0/pci8086,340e@7/pci1028,1f0a@0/sd@2,0
4. c3t3d0
/pci@0,0/pci8086,340e@7/pci1028,1f0a@0/sd@3,0
5. c3t4d0
/pci@0,0/pci8086,340e@7/pci1028,1f0a@0/sd@4,0
6. c3t5d0
/pci@0,0/pci8086,340e@7/pci1028,1f0a@0/sd@5,0
7. c3t6d0
/pci@0,0/pci8086,340e@7/pci1028,1f0a@0/sd@6,0
8. c3t7d0
/pci@0,0/pci8086,340e@7/pci1028,1f0a@0/sd@7,0
9. c3t8d0
/pci@0,0/pci8086,340e@7/pci1028,1f0a@0/sd@8,0
10. c3t9d0
/pci@0,0/pci8086,340e@7/pci1028,1f0a@0/sd@9,0
11. c3t10d0
/pci@0,0/pci8086,340e@7/pci1028,1f0a@0/sd@a,0
12. c3t11d0
/pci@0,0/pci8086,340e@7/pci1028,1f0a@0/sd@b,0
13. c3t12d0
/pci@0,0/pci8086,340e@7/pci1028,1f0a@0/sd@c,0
14. c3t13d0
/pci@0,0/pci8086,340e@7/pci1028,1f0a@0/sd@d,0
15. c3t14d0
/pci@0,0/pci8086,340e@7/pci1028,1f0a@0/sd@e,0
16. c3t15d0
/pci@0,0/pci8086,340e@7/pci1028,1f0a@0/sd@f,0
17. c3t16d0
/pci@0,0/pci8086,340e@7/pci1028,1f0a@0/sd@10,0
Specify disk (enter its number):

Just because

2010-01-20T10:52:00.000-08:00

Just because for some damn reason I can never remember this damn command. I mount an ISO all the time but I can never remember how. Sad.

mount -F hsfs `lofiadm -a /data/ISO/solaris.iso` /Solaris

DHCP

2010-01-18T06:20:00.000-08:00

When spending a lot of time trying to figure out why dhcp isn't working, there are several approches you can take. Me I took the snoop and find out what the packets say route. After much checking of config fines, double check mac address etc I found the problem. So for the future let me just say I have a new route I take trouble shooting dhcp.

first make sure the server is actually running.

2010-01-13T08:00:00.000-08:00

Is it just me or do programmers that play at being sysadmins makes stuff waaaaay to complicated ?

exhibit A:

a little script named "kill-apache", now as most of you know the way to stop apache is jsut /path/to/apache/bin/apachectl stop

one single command. This kill-apache script is 86 lines long. That is not counting the 6 libs it includes. I know I sound like a broken record but people really should keep stuff simple. So you login run a script "/opt/scripts/kill-apache" it calls in 6 libs, does a buch of checks and then it runs apachectl stop. The checks it is doing are really worthless. Like trying to figure out what server it is on. Stuff that wont matter to apache when it is killed.

apachectl stop

vs

200 + lines of code (counting the libs) to run the same command.

Adventures in sysadming

2009-12-06T18:19:00.000-08:00

Sometimes I have difficulty understanding why people do things. Reviewing some servers we recently took over I found some scripts that are real gems. For example this little guy is named isrunning-apache*. It does exactly what the name implies, here it is in its entirety**...

------------------------ #!/usr/bin/perl require("/opt/init/vars.pl"); $isrunning = 'no'; open(PSOUT, "/bin/ps ax | /bin/grep httpd |") || die("ERROR: cannot read ps output: $!\n"); while($line = ) { if($line =~ m/\/opt\/apache2\/bin\/httpd\ \-k\ start$/) { $isrunning = 'yes'; } } close(PSOUT); print "$isrunning\n"; -------------------------------------

Is it really too hard for people to type 'pgrep httpd'? Do people think this adds something? I mean it is not like he did something clever such as get the pid from the httpd.pid file and compare it to the running pid (have to watch out for zombies, eat your brain).

So kids, please... K.I.S.S.

* yes, there are other isrunning_xxx for the main apps.
** that require vars.pl is another 41 line perl script, that is not used in this script at all, just required.

2009-11-17T12:16:00.000-08:00

a bit more detailed test. This is the secondary EMC (i.e. Not in use) vs the new jbod. Both are NFS mounted on loki. This test is a webserer simulation, lots of opening/reading files, and the append log is the access log. Also the Jbod is now online with LACP failover on the nics going. Something that took us a long time to get working on EMCS was a single command line on the jbod.

EMC:
Flowop totals:
appendlog 2385ops/s 18.6mb/s 3.6ms/op 80us/op-cpu
closefile10 2375ops/s 0.0mb/s 0.0ms/op 15us/op-cpu
readfile10 2375ops/s 37.2mb/s 0.0ms/op 40us/op-cpu
openfile10 2376ops/s 0.0mb/s 0.4ms/op 88us/op-cpu
closefile9 2376ops/s 0.0mb/s 0.0ms/op 15us/op-cpu
readfile9 2376ops/s 37.6mb/s 0.0ms/op 40us/op-cpu
openfile9 2377ops/s 0.0mb/s 0.4ms/op 87us/op-cpu
closefile8 2377ops/s 0.0mb/s 0.0ms/op 15us/op-cpu
readfile8 2377ops/s 37.2mb/s 0.0ms/op 39us/op-cpu
openfile8 2378ops/s 0.0mb/s 0.4ms/op 87us/op-cpu
closefile7 2378ops/s 0.0mb/s 0.0ms/op 15us/op-cpu
readfile7 2378ops/s 37.2mb/s 0.0ms/op 39us/op-cpu
openfile7 2378ops/s 0.0mb/s 0.4ms/op 88us/op-cpu
closefile6 2378ops/s 0.0mb/s 0.0ms/op 15us/op-cpu
readfile6 2378ops/s 36.8mb/s 0.0ms/op 39us/op-cpu
openfile6 2378ops/s 0.0mb/s 0.4ms/op 87us/op-cpu
closefile5 2378ops/s 0.0mb/s 0.0ms/op 15us/op-cpu
readfile5 2378ops/s 37.1mb/s 0.0ms/op 39us/op-cpu
openfile5 2379ops/s 0.0mb/s 0.4ms/op 88us/op-cpu
closefile4 2379ops/s 0.0mb/s 0.0ms/op 15us/op-cpu
readfile4 2379ops/s 36.7mb/s 0.0ms/op 39us/op-cpu
openfile4 2381ops/s 0.0mb/s 0.4ms/op 88us/op-cpu
closefile3 2381ops/s 0.0mb/s 0.0ms/op 15us/op-cpu
readfile3 2381ops/s 37.4mb/s 0.0ms/op 40us/op-cpu
openfile3 2381ops/s 0.0mb/s 0.4ms/op 88us/op-cpu
closefile2 2381ops/s 0.0mb/s 0.0ms/op 15us/op-cpu
readfile2 2381ops/s 37.1mb/s 0.0ms/op 40us/op-cpu
openfile2 2382ops/s 0.0mb/s 0.4ms/op 89us/op-cpu
closefile1 2382ops/s 0.0mb/s 0.0ms/op 15us/op-cpu
readfile1 2372ops/s 37.4mb/s 0.0ms/op 41us/op-cpu
openfile1 2372ops/s 0.0mb/s 0.4ms/op 89us/op-cpu

IO Summary: 737516 ops 73721.4 ops/s, 23773/2385 r/w 390.7mb/s, 159uscpu/

JBOD:

Flowop totals:
appendlog 2880ops/s 22.4mb/s 0.8ms/op 44us/op-cpu
closefile10 2870ops/s 0.0mb/s 0.0ms/op 22us/op-cpu
readfile10 2870ops/s 45.1mb/s 0.0ms/op 34us/op-cpu
openfile10 2870ops/s 0.0mb/s 0.1ms/op 59us/op-cpu
closefile9 2870ops/s 0.0mb/s 0.0ms/op 22us/op-cpu
readfile9 2870ops/s 45.3mb/s 0.0ms/op 34us/op-cpu
openfile9 2871ops/s 0.0mb/s 0.1ms/op 58us/op-cpu
closefile8 2871ops/s 0.0mb/s 0.0ms/op 21us/op-cpu
readfile8 2871ops/s 44.8mb/s 0.0ms/op 34us/op-cpu
openfile8 2872ops/s 0.0mb/s 0.1ms/op 58us/op-cpu
closefile7 2872ops/s 0.0mb/s 0.0ms/op 21us/op-cpu
readfile7 2872ops/s 45.0mb/s 0.0ms/op 34us/op-cpu
openfile7 2873ops/s 0.0mb/s 0.1ms/op 58us/op-cpu
closefile6 2873ops/s 0.0mb/s 0.0ms/op 21us/op-cpu
readfile6 2873ops/s 44.9mb/s 0.0ms/op 33us/op-cpu
openfile6 2873ops/s 0.0mb/s 0.1ms/op 58us/op-cpu
closefile5 2873ops/s 0.0mb/s 0.0ms/op 21us/op-cpu
readfile5 2873ops/s 45.0mb/s 0.0ms/op 34us/op-cpu
openfile5 2875ops/s 0.0mb/s 0.1ms/op 58us/op-cpu
closefile4 2875ops/s 0.0mb/s 0.0ms/op 21us/op-cpu
readfile4 2875ops/s 44.8mb/s 0.0ms/op 33us/op-cpu
openfile4 2875ops/s 0.0mb/s 0.1ms/op 58us/op-cpu
closefile3 2875ops/s 0.0mb/s 0.0ms/op 21us/op-cpu
readfile3 2876ops/s 44.9mb/s 0.0ms/op 34us/op-cpu
openfile3 2877ops/s 0.0mb/s 0.1ms/op 58us/op-cpu
closefile2 2877ops/s 0.0mb/s 0.0ms/op 22us/op-cpu
readfile2 2877ops/s 44.7mb/s 0.0ms/op 34us/op-cpu
openfile2 2878ops/s 0.0mb/s 0.1ms/op 58us/op-cpu
closefile1 2878ops/s 0.0mb/s 0.0ms/op 22us/op-cpu
readfile1 2868ops/s 44.9mb/s 0.0ms/op 34us/op-cpu
openfile1 2870ops/s 0.0mb/s 0.1ms/op 60us/op-cpu

IO Summary: 891131 ops 89077.1 ops/s, 28726/2880 r/w 472.1mb/s, 120uscpu/op

Quick mkfile test (yes I know it is not very scentific)

2009-11-17T12:15:00.000-08:00

V Test is to create a 10GB file.

HEL: Dell 1950 Local 15k sas drive: OpenSolaris 2009/06
[root@hel:]% time mkfile 10g bigfile01

real 1m17.162s
user 0m0.032s
sys 0m6.726s

LOKI: Dell 2950 Local 10k sas drive: OpenSolaris B96
[root@loki:]% time mkfile 10g bigfile01

real 3m3.819s
user 0m0.039s
sys 0m9.532s

NFS03: Dell R610 local 15k sas drive: OpenSolaris 2009/06
root@webnfs03:/# time mkfile 10g bigfile01

real 1m42.640s
user 0m0.039s
sys 0m4.893s

NFS03: Dell R610 using MD1120 Mounted as RaidZ (zfs): Opensoaris 2009/06
root@webnfs03:/data# time mkfile 10g bigfile02

real 0m10.508s
user 0m0.026s
sys 0m4.497s

Holy crap sticks. The jbod is 10 times faster then the local disk.