tech.gate.io blog

Solaris 10 system crashed, root filesystem mounted read only

Tags: solaris ufs mirror

this happened to me two weeks ago, Solaris10 system crashed, rebooted automatically, networking seemed to be up, but no possibility to log in with ssh, ftp...

login to ALOM

console -f

logged in to system

I've found the following situation:

>df
/                  (/pseudo/md@0:0,10,blk):22226916 blocks  2070155 files
/devices           (/devices          ):       0 blocks        0 files
/system/contract   (ctfs              ):       0 blocks 2147483641 files
/proc              (proc              ):       0 blocks    29991 files
/etc/mnttab        (mnttab            ):       0 blocks        0 files
/etc/svc/volatile  (swap              ):14964432 blocks   565745 files
/system/object     (objfs             ):       0 blocks 2147483486 files
/platform/sun4v/lib/libc_psr.so.1(/condis1:/-->id
latform/SUNW,Sun-Fire-T1000/lib/libc_psr/libc_psr_hwcap1.so.1):22226916 blocks  2070155 files
/platform/sun4v/lib/sparcv9/libc_psr.so.1(/platform/SUNW,Sun-Fire-T1000/lib/sparcv9/libc_psr/libc_psr_hwcap1.so.1):22226916 blocks  2070155 files

root filesystem mounted read only, no /tmp or /var was mounted

at first I rebooted the system:

>shutdown -Fr
/usr/sbin/shutdown: /tmp/sh1560: cannot create

>init 6
Jun 15 10:16:35 su: 'su root' succeeded for root on /dev/???

the tmp file could not be placed because of the read only /tmp, rebooted with init 6

Hostname: xxxx
The / file system (/dev/md/rdsk/d10) is being checked.
ERROR: svc:/system/filesystem/usr:default failed to mount /  (see 'svcs -x' for details)
Jun 15 10:19:36 svc.startd[8]: svc:/system/filesystem/usr:default: Method "/lib/svc/method/fs-usr" failed with exit status 95.
Jun 15 10:19:36 svc.startd[8]: system/filesystem/usr:default failed fatally: transitioned to maintenance (see 'svcs -xv' for details)
Requesting System Maintenance Mode
(See /lib/svc/share/README for more information.)
Console login service(s) cannot run

Root password for system maintenance (control-d to bypass):
single-user privilege assigned to /dev/console.
Entering System Maintenance Mode

"mount" still showed a read only root fs

>svcs -a
STATE          STIME    FMRI
disabled       10:18:19 svc:/network/iscsi_initiator:default
disabled       10:18:19 svc:/system/device/mpxio-upgrade:default
disabled       10:18:20 svc:/network/ipfilter:default
disabled       10:18:21 svc:/network/inetd-upgrade:default
disabled       10:18:21 svc:/network/rpc/keyserv:default
disabled       10:18:21 svc:/network/rpc/nisplus:default
disabled       10:18:21 svc:/network/nis/server:default
disabled       10:18:21 svc:/network/nis/client:default
disabled       10:18:21 svc:/network/dns/client:default
disabled       10:18:21 svc:/network/ldap/client:default
disabled       10:18:22 svc:/network/nfs/client:default
disabled       10:18:22 svc:/application/print/server:default
disabled       10:18:22 svc:/network/ntp:default
disabled       10:18:22 svc:/system/auditd:default
disabled       10:18:22 svc:/system/patch-finish:delete
disabled       10:18:22 svc:/system/rcap:default
disabled       10:18:23 svc:/network/rpc/bootparams:default
disabled       10:18:23 svc:/network/samba:default
disabled       10:18:23 svc:/network/winbind:default
disabled       10:18:23 svc:/network/wins:default
disabled       10:18:23 svc:/network/nfs/server:default
disabled       10:18:23 svc:/network/rarp:default
disabled       10:18:23 svc:/network/dhcp-server:default
disabled       10:18:24 svc:/application/management/webmin:default
disabled       10:18:24 svc:/application/database/postgresql:version_81
disabled       10:18:24 svc:/application/database/postgresql:version_82
disabled       10:18:24 svc:/application/database/postgresql:version_82_64bit
disabled       10:18:24 svc:/application/gdm2-login:default
disabled       10:18:25 svc:/network/dns/server:default
disabled       10:18:25 svc:/network/routing/legacy-routing:ipv4
disabled       10:18:25 svc:/network/routing/legacy-routing:ipv6
disabled       10:18:25 svc:/network/routing/ndp:default
disabled       10:18:25 svc:/network/routing/rdisc:default
disabled       10:18:25 svc:/network/ipv6-forwarding:default
disabled       10:18:25 svc:/network/routing/ripng:default
disabled       10:18:25 svc:/network/routing/zebra:quagga
disabled       10:18:25 svc:/network/routing/ripng:quagga
disabled       10:18:25 svc:/network/routing/route:default
disabled       10:18:25 svc:/network/ipv4-forwarding:default
disabled       10:18:25 svc:/network/routing/rip:quagga
disabled       10:18:25 svc:/network/routing/ospf:quagga
disabled       10:18:25 svc:/network/routing/ospf6:quagga
disabled       10:18:25 svc:/network/routing/bgp:quagga
disabled       10:18:26 svc:/network/security/kadmin:default
disabled       10:18:26 svc:/network/security/krb5kdc:default
disabled       10:18:28 svc:/network/ipmievd:default
disabled       10:18:29 svc:/network/nis/passwd:default
disabled       10:18:29 svc:/network/nis/update:default
disabled       10:18:29 svc:/network/nis/xfr:default
disabled       10:18:29 svc:/network/http:apache2
disabled       10:18:29 svc:/network/apocd/udp:default
disabled       10:18:30 svc:/network/slp:default
disabled       10:18:30 svc:/platform/sun4u/efdaemon:default
disabled       10:18:31 svc:/ldoms/vntsd:default
disabled       10:18:31 svc:/system/consadm:default
disabled       10:18:32 svc:/system/iscsitgt:default
disabled       10:18:32 svc:/system/sar:default
disabled       10:18:33 svc:/system/cluster/scsymon-srv:default
disabled       10:18:33 svc:/system/cluster/sc_restarter:default
disabled       10:18:33 svc:/system/cluster/sc_ng_zones:default
online         10:18:18 svc:/system/svc/restarter:default
online         10:18:20 svc:/network/pfil:default
online         10:18:20 svc:/network/loopback:default
online         10:18:24 svc:/system/installupdates:default
online         10:18:25 svc:/milestone/name-services:default
online         10:18:27 svc:/network/physical:default
online         10:18:30 svc:/milestone/network:default
online         10:18:30 svc:/system/identity:node
online         10:18:36 svc:/system/metainit:default
online         10:18:37 svc:/system/filesystem/root:default
online         10:18:37 svc:/system/boot-archive:default
online         10:18:37 svc:/system/scheduler:default
online         10:19:37 svc:/network/service:default
offline        10:18:19 svc:/system/cluster/cl_boot_check:default
offline        10:18:19 svc:/system/cluster/scmountdev:default
offline        10:18:19 svc:/system/device/local:default
offline        10:18:19 svc:/system/filesystem/minimal:default
offline        10:18:19 svc:/system/sysevent:default
offline        10:18:19 svc:/system/device/fc-fabric:default
offline        10:18:19 svc:/milestone/devices:default
offline        10:18:19 svc:/system/identity:domain
offline        10:18:20 svc:/system/cluster/loaddid:default
offline        10:18:20 svc:/system/coreadm:default
offline        10:18:20 svc:/network/multipath:cluster
offline        10:18:20 svc:/system/cryptosvc:default
offline        10:18:20 svc:/network/initial:default
offline        10:18:20 svc:/system/manifest-import:default
offline        10:18:20 svc:/network/routing-setup:default
offline        10:18:20 svc:/system/cluster/bootcluster:default
offline        10:18:20 svc:/system/cluster/initdid:default
offline        10:18:20 svc:/system/keymap:default
offline        10:18:20 svc:/milestone/single-user:default
offline        10:18:20 svc:/system/filesystem/local:default
offline        10:18:20 svc:/system/cluster/scvxinstall:default
offline        10:18:21 svc:/system/sysidtool:net
offline        10:18:21 svc:/network/rpc/bind:default
offline        10:18:21 svc:/system/sysidtool:system
offline        10:18:21 svc:/milestone/sysconfig:default
offline        10:18:21 svc:/network/inetd:default
offline        10:18:21 svc:/system/mdmonitor:default
offline        10:18:21 svc:/system/cluster/sc_failfast:default
offline        10:18:21 svc:/system/cluster/clexecd:default
offline        10:18:21 svc:/system/cluster/globaldevices:default
offline        10:18:21 svc:/system/cluster/gdevsync:default
offline        10:18:21 svc:/network/nfs/status:default
offline        10:18:21 svc:/network/nfs/nlockmgr:default
offline        10:18:22 svc:/network/nfs/cbd:default
offline        10:18:22 svc:/network/nfs/mapid:default
offline        10:18:22 svc:/system/filesystem/autofs:default
offline        10:18:22 svc:/system/system-log:default
offline        10:18:22 svc:/system/utmp:default
offline        10:18:22 svc:/network/smtp:sendmail
offline        10:18:22 svc:/system/cron:default
offline        10:18:22 svc:/system/name-service-cache:default
offline        10:18:22 svc:/system/rmtmpfiles:default
offline        10:18:22 svc:/system/pools:default
offline        10:18:22 svc:/system/resource-mgmt:default
offline        10:18:22 svc:/milestone/multi-user:default
offline        10:18:22 svc:/system/console-login:default
offline        10:18:23 svc:/system/cluster/cl-svc-enable:default
offline        10:18:23 svc:/application/management/seaport:default
offline        10:18:23 svc:/application/management/snmpdx:default
offline        10:18:23 svc:/application/management/dmi:default
offline        10:18:23 svc:/network/ssh:default
offline        10:18:23 svc:/milestone/multi-user-server:default
offline        10:18:23 svc:/application/font/fc-cache:default
offline        10:18:23 svc:/application/graphical-login/cde-login:default
offline        10:18:24 svc:/system/cluster/ql_upgrade:default
offline        10:18:24 svc:/system/cluster/mountgfs:default
offline        10:18:24 svc:/system/cluster/sc_zones:default
offline        10:18:24 svc:/system/zones:default
offline        10:18:24 svc:/application/management/wbem:default
offline        10:18:24 svc:/application/management/sma:default
offline        10:18:24 svc:/application/cde-printinfo:default
offline        10:18:24 svc:/application/print/ipp-listener:default
offline        10:18:30 svc:/platform/sun4v/drd:default
offline        10:18:31 svc:/platform/sun4v/efdaemon:default
offline        10:18:31 svc:/system/filesystem/volfs:default
offline        10:18:32 svc:/system/sac:default
offline        10:18:32 svc:/system/dumpadm:default
offline        10:18:32 svc:/system/fmd:default
offline        10:18:32 svc:/system/webconsole:console
offline        10:18:32 svc:/system/pools/dynamic:default
offline        10:18:32 svc:/system/basicreg:default
offline        10:18:32 svc:/system/picl:default
offline        10:18:32 svc:/application/stosreg:default
offline        10:18:32 svc:/application/psncollector:default
offline        10:18:32 svc:/system/cluster/cl-ccra:default
offline        10:18:32 svc:/system/cluster/cl-event:default
offline        10:18:32 svc:/system/cluster/cl-eventlog:default
offline        10:18:33 svc:/system/cluster/scprivipd:default
offline        10:18:33 svc:/system/cluster/scdpm:default
offline        10:18:33 svc:/system/cluster/rpc-pmf:default
offline        10:18:33 svc:/system/cluster/scslm:default
offline        10:18:33 svc:/system/cluster/rpc-fed:default
offline        10:18:33 svc:/system/cluster/pnm:default
offline        10:18:33 svc:/system/cluster/rgm:default
offline        10:18:33 svc:/system/cluster/cl-svc-cluster-milestone:default
offline        10:18:33 svc:/system/cluster/clusterdata:default
offline        10:18:33 svc:/system/cluster/ql_rgm:default
offline        10:18:33 svc:/system/cluster/scslmclean:default
offline        10:18:33 svc:/application/management/common-agent-container-1:default
offline        10:18:33 svc:/system/cluster/sc_svtag:default
offline        10:18:34 svc:/system/cluster/sckeysync:default
offline        10:18:34 svc:/application/print/ppd-cache-update:default
maintenance    10:19:36 svc:/system/filesystem/usr:default
uninitialized  10:18:20 svc:/network/rpc/meta:default
uninitialized  10:18:22 svc:/network/rpc/gss:default
uninitialized  10:18:23 svc:/application/x11/xfs:default
uninitialized  10:18:23 svc:/application/font/stfsloader:default
uninitialized  10:18:24 svc:/network/rpc/rstat:default
uninitialized  10:18:24 svc:/application/print/rfc1179:default
uninitialized  10:18:25 svc:/network/rpc/cde-calendar-manager:default
uninitialized  10:18:26 svc:/network/rpc/cde-ttdbserver:tcp
uninitialized  10:18:26 svc:/network/rpc/ocfserv:default
uninitialized  10:18:26 svc:/network/rpc/smserver:default
uninitialized  10:18:26 svc:/network/rpc/mdcomm:default
uninitialized  10:18:26 svc:/network/rpc/metamed:default
uninitialized  10:18:26 svc:/network/rpc/metamh:default
uninitialized  10:18:26 svc:/network/rpc/rex:default
uninitialized  10:18:26 svc:/network/rpc/rusers:default
uninitialized  10:18:26 svc:/network/rpc/spray:default
uninitialized  10:18:26 svc:/network/rpc/wall:default
uninitialized  10:18:26 svc:/network/security/ktkt_warn:default
uninitialized  10:18:26 svc:/network/security/krb5_prop:default
uninitialized  10:18:26 svc:/network/swat:default
uninitialized  10:18:26 svc:/network/cde-spc:default
uninitialized  10:18:27 svc:/network/tname:default
uninitialized  10:18:27 svc:/network/telnet:default
uninitialized  10:18:27 svc:/network/nfs/rquota:default
uninitialized  10:18:29 svc:/network/uucp:default
uninitialized  10:18:29 svc:/network/chargen:dgram
uninitialized  10:18:29 svc:/network/chargen:stream
uninitialized  10:18:29 svc:/network/daytime:dgram
uninitialized  10:18:29 svc:/network/daytime:stream
uninitialized  10:18:29 svc:/network/discard:dgram
uninitialized  10:18:29 svc:/network/discard:stream
uninitialized  10:18:29 svc:/network/echo:dgram
uninitialized  10:18:29 svc:/network/echo:stream
uninitialized  10:18:30 svc:/network/time:dgram
uninitialized  10:18:30 svc:/network/time:stream
uninitialized  10:18:30 svc:/network/ftp:default
uninitialized  10:18:30 svc:/network/comsat:default
uninitialized  10:18:30 svc:/network/finger:default
uninitialized  10:18:30 svc:/network/login:eklogin
uninitialized  10:18:30 svc:/network/login:klogin
uninitialized  10:18:30 svc:/network/login:rlogin
uninitialized  10:18:30 svc:/network/rexec:default
uninitialized  10:18:30 svc:/network/shell:default
uninitialized  10:18:30 svc:/network/shell:kshell
uninitialized  10:18:30 svc:/network/talk:default
uninitialized  10:18:32 svc:/network/rpc-100235_1/rpc_ticotsord:default
uninitialized  10:18:32 svc:/network/stdiscover:default
uninitialized  10:18:32 svc:/network/stlisten:default
uninitialized  10:18:32 svc:/network/rpc/scadmd:default
uninitialized  10:18:32 svc:/network/rpc/scrcmd:default
uninitialized  10:18:32 svc:/network/sccheckd:default
uninitialized  10:18:33 svc:/network/rpc/metacld:default

most services are offline or uninitialized, no wonder with no writeable var and tmp

>prtdiag
picl_initialize failed: Daemon not responding

hummmm

fsck -y /
** /dev/md/rdsk/d10
BAD SUPERBLOCK AT BLOCK 16: MAGIC NUMBER WRONG

LOOK FOR ALTERNATE SUPERBLOCKS WITH MKFS?  yes

FOUND ALTERNATE SUPERBLOCK 32 WITH MKFS

USE ALTERNATE SUPERBLOCK?  yes

FOUND ALTERNATE SUPERBLOCK AT 32 USING MKFS
If filesystem was created with manually-specified geometry, using
auto-discovered superblock may result in irrecoverable damage to
filesystem and user data.

CANCEL FILESYSTEM CHECK?  yes

Please verify that the indicated block contains a proper
superblock for the filesystem (see fsdb(1M)).

FSCK was running in YES mode.  If you wish to run in that mode using
the alternate superblock, run `fsck -y -o b=32 /dev/md/rdsk/d10'.
@uscondis1:/-->fsck -y -o b=32 /dev/md/rdsk/d10
Alternate super block location: 32.
** /dev/md/rdsk/d10
** Last Mounted on
** Phase 1 - Check Blocks and Sizes

** Phase 2 - Check Pathnames
UNALLOCATED  I=6460  OWNER=root MODE=0
SIZE=0 MTIME=Jan  1 01:00 1970
NAME=/opt/tivoli/lcf/dat/1/LCFNEW/AMW/logs/msg_dmxengine.log

REMOVE DIRECTORY ENTRY FROM I=5828?  yes

** Phase 3a - Check Connectivity
** Phase 3b - Verify Shadows/ACLs
** Phase 4 - Check Reference Counts
UNREF FILE  I=5857  OWNER=bbuser MODE=100644
SIZE=8 MTIME=May 10 03:58 2009
RECONNECT?  yes

** Phase 5 - Check Cylinder Groups

CORRECT BAD CG SUMMARIES FOR CG 0?  yes

..
..

a lot of damaged Inodes later, system seems to be fine, run a second fsck, no problem found

alright, reboot then

dupdidu...
.............

........................

ok system is up, still read only mounted root filesystem, forcing to remount it rw didn't work either

so lets have a look at the ufs mirror, I should have checked it earlier

>metastat
d50: Mirror
    Submirror 0: d51
      State: Needs maintenance
    Submirror 1: d52
      State: Needs maintenance
    Pass: 1
    Read option: roundrobin (default)
    Write option: parallel (default)
    Size: 8405376 blocks (4.0 GB)

d51: Submirror of d50
    State: Needs maintenance
    Invoke: metareplace d50 /dev/dsk/c3t5000C50007AD47E7d0s5 
    Size: 8405376 blocks (4.0 GB)
    Stripe 0:
        Device                             Start Block  Dbase        State Reloc Hot Spare
        /dev/dsk/c3t5000C50007AD47E7d0s5          0     No     Maintenance   Yes


d52: Submirror of d50
    State: Needs maintenance
    Invoke: metasync d50
    Size: 8405376 blocks (4.0 GB)
    Stripe 0:
        Device                             Start Block  Dbase        State Reloc Hot Spare
        /dev/dsk/c3t5000C50007AD255Fd0s5          0     No            Okay   Yes


d20: Mirror
    Submirror 0: d21
      State: Needs maintenance
    Submirror 1: d22
      State: Needs maintenance
    Pass: 1
    Read option: roundrobin (default)
    Write option: parallel (default)
    Size: 8405376 blocks (4.0 GB)

d21: Submirror of d20
    State: Needs maintenance
    Invoke: metareplace d20 /dev/dsk/c3t5000C50007AD47E7d0s1 
    Size: 8405376 blocks (4.0 GB)
    Stripe 0:
        Device                             Start Block  Dbase        State Reloc Hot Spare
        /dev/dsk/c3t5000C50007AD47E7d0s1          0     No     Maintenance   Yes
d22: Submirror of d20
    State: Needs maintenance
    Invoke: metasync d20
    Size: 8405376 blocks (4.0 GB)
    Stripe 0:
        Device                             Start Block  Dbase        State Reloc Hot Spare
        /dev/dsk/c3t5000C50007AD255Fd0s1          0     No            Okay   Yes


d40: Mirror
    Submirror 0: d41
      State: Needs maintenance
    Submirror 1: d42
      State: Needs maintenance
    Pass: 1
    Read option: roundrobin (default)
    Write option: parallel (default)
    Size: 37752960 blocks (18 GB)

d41: Submirror of d40
    State: Needs maintenance
    Invoke: metasync d40
    Size: 37752960 blocks (18 GB)
    Stripe 0:
        Device                             Start Block  Dbase        State Reloc Hot Spare
        /dev/dsk/c3t5000C50007AD47E7d0s4          0     No            Okay   Yes


d42: Submirror of d40
    State: Needs maintenance
    Invoke: metasync d40
    Size: 37752960 blocks (18 GB)
    Stripe 0:
        Device                             Start Block  Dbase        State Reloc Hot Spare
        /dev/dsk/c3t5000C50007AD255Fd0s4          0     No            Okay   Yes


d30: Mirror
    Submirror 0: d31
      State: Needs maintenance
    Submirror 1: d32
      State: Needs maintenance
    Pass: 1
    Read option: roundrobin (default)
    Write option: parallel (default)
    Size: 1058304 blocks (516 MB)

d31: Submirror of d30
    State: Needs maintenance
    Invoke: metareplace d30 /dev/dsk/c3t5000C50007AD47E7d0s3 
    Size: 1058304 blocks (516 MB)
    Stripe 0:
        Device                             Start Block  Dbase        State Reloc Hot Spare
        /dev/dsk/c3t5000C50007AD47E7d0s3          0     No     Maintenance   Yes


d32: Submirror of d30
    State: Needs maintenance
    Invoke: metasync d30
    Size: 1058304 blocks (516 MB)
    Stripe 0:
        Device                             Start Block  Dbase        State Reloc Hot Spare
        /dev/dsk/c3t5000C50007AD255Fd0s3          0     No            Okay   Yes


d10: Mirror
    Submirror 0: d11
      State: Needs maintenance
    Submirror 1: d12
      State: Needs maintenance
    Pass: 1
    Read option: roundrobin (default)
    Write option: parallel (default)
    Size: 37752960 blocks (18 GB)

d11: Submirror of d10
    State: Needs maintenance
    Invoke: metareplace d10 /dev/dsk/c3t5000C50007AD47E7d0s0 
    Size: 37752960 blocks (18 GB)
    Stripe 0:
        Device                             Start Block  Dbase        State Reloc Hot Spare
        /dev/dsk/c3t5000C50007AD47E7d0s0          0     No     Maintenance   Yes


d12: Submirror of d10
    State: Needs maintenance
    Invoke: metasync d10
    Size: 37752960 blocks (18 GB)
    Stripe 0:
        Device                             Start Block  Dbase        State Reloc Hot Spare
        /dev/dsk/c3t5000C50007AD255Fd0s0          0     No            Okay   Yes
Device Relocation Information:
Device                           Reloc  Device ID
/dev/dsk/c3t5000C50007AD255Fd0   Yes    id1,sd@n5000c50007ad255f
/dev/dsk/c3t5000C50007AD47E7d0   Yes    id1,sd@n5000c50007ad47e7

grr I hate solaris ufs mirrors

the suggested commands didn't work, I'm asking my self why the hell solaris wont come up with an unsynchronised mirror, support told us later, that there was a 3 second scsi io error 2 weeks ago, the mirror broke up that time, and they said this must be a bug, go and update to latest solaris patchcluster -.-

the solution for this problem was:

for every mirror, I ran:

>metareplace -e mirrorname slicename

for example:
>metareplace -e d50 c3t5000C50007AD47E7d0s5
d50: device c3t5000C50007AD47E7d0s5 is enabled
metareplace: uscondis1: /etc/lvm/md.cf.new: Read-only file system


>metastat
d50: Mirror
    Submirror 0: d51
      State: Resyncing
    Submirror 1: d52
      State: Okay
    Resync in progress: 44 % done
    Pass: 1
    Read option: roundrobin (default)
    Write option: parallel (default)
    Size: 8405376 blocks (4.0 GB)

d51: Submirror of d50
    State: Resyncing
    Size: 8405376 blocks (4.0 GB)
    Stripe 0:
        Device                             Start Block  Dbase        State Reloc Hot Spare
        /dev/dsk/c3t5000C50007AD47E7d0s5          0     No       Resyncing   Yes


d52: Submirror of d50
    State: Okay
    Size: 8405376 blocks (4.0 GB)
    Stripe 0:
        Device                             Start Block  Dbase        State Reloc Hot Spare
        /dev/dsk/c3t5000C50007AD255Fd0s5          0     No            Okay   Yes

reboot - system was fine

still wondering how it's possible that the system could start the maintenance mode without /var or /tmp

cheers

Permalink

← Clone Bootdisk / reinstall grub

LAN failover solution for HPUX without serviceguard →

Posted by hphmrybo on Mon 07 of June, 2010 14:18 CEST

vvengopk zxvkijxx http://hesuqbrf.com qsruvzit jezcmhhe url=http://qgqrrtac.comwmlraqkb/url

Posted by orgdhyci on Sat 31 of July, 2010 01:54 CEST

url=http://sygsjwib.comnwudyusx/url pukiskzn http://ieomeyiw.com rqohdzrg qlfuhijx rhqgjibm

Posted by pjafppbn on Sat 31 of July, 2010 01:09 CEST

abnwhadh fsrytlcx http://uyqfkeic.com angqeiep kaarjsml url=http://nrvplzme.comnzazyobl/url

Posted by diavvflv on Sat 31 of July, 2010 01:08 CEST

thuwkcod http://kxaehhkd.com wshxlvmb gqzkczoq ftwfhbot url=http://byvihgno.comchzwjoyy/url

Posted by ozpaegsb on Sat 24 of July, 2010 00:59 CEST

nirabsza http://mxdlzojx.com rdsuwpiy jqlhlhta cyqktqlh url=http://ieeacsho.comvzbulckr/url

Name
Title *

Comment *	Add Comment

Enter the code you see above *	Try another code

tech.gate.io blog

Solaris 10 system crashed, root filesystem mounted read only

34 comments

Post new comment

Sidebar

Wiki

Sidebar

Most popular tags

Last blog posts

Last blog post comments

Feeds List