Recently, I attempted to upgrade a 10G database on Linux for 11.2. Before upgrading production I ran the procedure on a test database (on another server) with no problems. However, in the production environment the catupgrd.sql script failed. This happened over the weekend so I opened a severity 1 SR and began working with Oracle. I sent all the required information but Oracle was unable to get me past the issue. I only had about 8 hours before needing to rollback because this is an 8TB database. The script has gotten far enough to require an RMAN restore. This restore/recovery took about 60 hours to complete so this was quite painful. I continued working with Oracle on the SR to try and get to the root of the problem. It turns out that natively compiled PL/SQL (which is being used in this environment) stored the compiled binaries in /dev/shm on Linux. There was an error in the alertlog referring to /dev/shm but Oracle did not identify the cause within the 8 hour window I had for the upgrade. Turns out that the other machine that was used to test the upgrade had /dev/shm enabled but the production machine did not.
Takeways:
- Make sure /dev/shm is enabled when upgrading a Linux environment that uses natively compiled stored procedures
- Consistency at the OS level in your environment is a must