I'm involved in a project to migrate a 4TB database from HP/UX 11 and Oracle 9i to a brand-new Sun M5000 server with Oracle 126.96.36.199. This database suffers insert transactions in the order of 70 tx/sec. The daily redo production is in the order of 45GB. Management reports are also run with great vigour (i.e. large volumes of disk read IOPS). Two further tiny instances (5-7GB each) also live in the same environment.
The original plan was to install Solaris 10 on the new server and create a big ZFS pool on the san, as proposed by Oracle Sales. However, doubts have arisen as to the performance of ZFS with Oracle databases, and we now lean towards using UFS for the database files. All discussions and white papers that I have been able to find on the subject stress to closely follow the upgrade path, as ZFS is continuously being improved still. Some blogs give pointers on how to make ZFS perform "almost the same as UFS", which sounds to me as a lot of extra effort for no gain. I struggle to find any validation for choosing ZFS over UFS.
Today, the boss was told by a relation who used to work for Sun that that relation would no longer install boxes with UFS. He would also enable direct IO instead of totally relying on ZFS. The SAN disks should according to this relation be presented as raw disks, rather than striped-and-mirrored LUNs, to be RAIDed in ZFS. Apparently there are desirable features in ZFS that make this worthwhile. It should be noted that the SAN is (almost) completely dedicated to this one database machine and has block copy capabilities, built-in raid, etc.
To me it seems a bit back-to-front to disable the SAN functionality, effectively turning it into an expensive external disk array, and at the same time shifting all the work that the SAN would have done to the database machine CPU where it competes for resources with the Oracle instances. What advantages, if any, exist that make using ZFS in this way is preferable over UFS? Do you have any experience with it?
The Solaris version was bought before Oracle certified 188.8.131.52 on Solaris 11, but now it seems silly not to upgrade Solaris before this system goes life. It will quite possibly not be able to be upgraded any time soon, possibly not until after Oracle 14x is released.. ;) The same relation however also insisted that "there are certification issues with Solaris 11" and he would never install Oracle 11g database on Solaris 11. However, MOS clearly shows that 184.108.40.206 is fully certified on Solaris 11. Do you happen to know what issues could exist that pre-empt the use of Solaris 11, even if that might mean that the client will be on Solaris 10 for the next decade?
I would like to hear about your experiences and thoughts.