We are running into a strange issue where one of our RAC cluster nodes cannot start an instance but the other node in the cluster has no issues. They are both the same as far as hardware and software config goes. Here are the errors we are getting:
SQL> startup
ORA-27154: post/wait create failed
ORA-27300: OS system dependent operation:semget failed with status: 28
ORA-27301: OS failure message: No space left on device
ORA-27302: failure occurred at: sskgpcreates
We found an oracle doc for these errors that said to bump up kernel.sem which we did but it didn't help.
We went from 250 32000 100 256 to 250 32000 100 384. We have many other databases on the cluster so did not want to bump it any higher. If we bump it up more is there a chance we could crash the node? We have a non-rac aware app that is hard coded to the node so we need to avoid crashing it.
Anyone have any ideas as to why the two nodes are out of sync and what values can we up the parameters to that is safe?
Thank you for any help.
Greg