About Me

My photo
I am Suresh Chinta, working on SAP HANA Cloud & SAP BTP Cloud/ AWS/Azure cloud consultant.I have experience in SAP Basis/Netweaver , S4HANA Cloud implementations / Support. I'm certified Microsoft Azure cloud & AWS professional. I have started this blog to share my knowledge with all those who are interested to learn & enhance their career.

Wednesday, May 13, 2020

SAP SYSTEM STARTUP ISSUES & SOLUTIONS





SAP System startup Problems:

Two places you need to check: EventViewer (Application and System logs) and the SAP Management Console (MMC).
Event Viewer can provide useful information and it may help you pinpoint where the problem resides. The SAP MMC gives you the ability to visually see the system status (green, yellow or red lights), view the work processes status and view the developer traces, which are stored in the "work" directory. Example: /usr/sap/TST/DVEBMGS00/work.
For a central SAP instance to start successfully, both the message server and the dispatcher need to start. If one of them or both fail to start, users cannot log in to the system. The following scenarios will illustrate possible causes of why an SAP instance might not start and the reason of the message:
"DISPATCHER EMERGENCY SHUTDOWN ".
Developer Traces:
dev_disp Dispatcher developer trace
dev_ms Message Server developer trace
dev_wp0 Work process 0 developer trace
The "services" file, which contains TCP and UDP services and their respective port numbers. This plain-text configuration file is located under winnt/system32/drivers/etc.
Windows Task Manager (TASKMGR.exe), Event Viewer (EVENTVWR.exe).
Dispatcher Monitor (DPMON.exe), which is located under /usr/sap//sys/exe/run. Database logs.

1. Dispatcher does not start due to a port conflict
No work processes (disp+work.exe) exist in Task Manager.
Dispatcher shows status "stopped" in the SAP MMC.
Errors found in "dev_disp":
*** ERROR => NiIBind: service sapdp00 in use [nixxi.c 3936]
*** ERROR => NiIDgBind: NiBind (rc=-4) [LOG Q0I=> NiPBind: bind (10048: WSAEADDRINUSE: Address already in use) [ninti.c 1488]
nixxi.c 3505]
*** ERROR => DpCommInit: NiDgBind [dpxxdisp.c 7326]
*** DP_FATAL_ERROR => DpSapEnvInit: DpCommInit
*** DISPATCHER EMERGENCY SHUTDOWN ***
Problem Analysis
I highlighted the keywords in the error messages above: Address already in use Service sapdp00 in use The TCP port number assigned in the "services" file is being occupied by another application. Due to the conflict, the dispatcher shuts down.
Solution
If your server has a firewall client, disable it and attempt to start the SAP instance again.
If the instance starts successfully you can enable the client firewall back again.
If there is no firewall client at all, or if disabling it did not resolve the problem, edit the "services" file and check what port the appropriate "sapdp" is using.
If the instance number is 00, look for sapdp00. If the instance number is 01 look for sapdp01 and so on. You can use the following OS command to help you resolve port conflicts:
netstat -p TCP There are also utilities on the Internet that can help you list all the TCP and UDP ports a system is using.

2: Dispatcher dies due to a database connection problem
database connections.
No work processes
SAP MMC -> WP Table shows all processes as "ended".
Errors found in "dev_disp":
C setuser 'tst' failed -- connect terminated
C failed to establish conn. 0
M ***LOG R19=> tskh_init, db_connect (DB-Connect 000256) [thxxhead.c 1102]
M in_ThErrHandle: 1
M *** ERROR => tskh_init: db_connect (step 1, th_errno 13, action 3, level 1) [thxxhead.c 8437]
*** ERROR => W0 (pid 2460) died [dpxxdisp.c 11651]
*** ERROR => W1 (pid 2468) died [dpxxdisp.c 11651]
*** ERROR => W2 (pid 2476) died [dpxxdisp.c 11651]. . .
*** ERROR => W11 (pid 2552) died [dpxxdisp.c 11651]
*** ERROR => W12 (pid 2592) died [dpxxdisp.c 11651]
my types changed after wp death/restart 0xbf --> 0x80
*** DP_FATAL_ERROR => DpEnvCheck: no more work processes
*** DISPATCHER EMERGENCY SHUTDOWN ***
DpModState: change server state from STARTING to SHUTDOWN
Problem Analysis
A connection to the database could not be established because either the SQL login specified in parameter "dbs/mss/schema" is set incorrectly or the SQL login was deleted from the database server. This parameter needs to be set in the DEFAULT.pfl system profile (under /usr/sap//sys/profile). In the messages above, we see that the SQL login 'tst' is expected but it does not exist at the database level.
Solution
Set the entry to the appropriate database owner. If the system is based on Basis <= 4.6 or if the system was upgraded from 4.x to 4.7 the database owner should be "dbo". But, if the system was installed from scratch and it's based on the Web AS 6.x the database owner should match the SID name in lower case. Example: if the SID is TST then the database owner should be "tst". If the parameter is set correctly in the DEFAULT.pfl profile check at the database level if the SQL login exists. If it doesn't, create it and give it database ownership to the .

3: SAP does not start at all: no message server and no dispatcher
The message server and the dispatcher do not start at all in the SAP MMC. The following error when trying to view the developer traces within the SAP MMC: The network path was not found. No new developer traces written to disk (under the "work" directory.)
Problem Analysis
The network shares "saploc" and "sapmnt" do not exist. That explains the "network path not found" message when attempting to view the developer traces within the SAP MMC.
Solution
Re-create the "saploc" and "sapmnt" network shares. Both need to be created on the /usr/sap directory

4: Users get "No logon possible" messages

Work processes start but no logins are possible.
Users get the login screen but the system does not log them in. Instead, they get this error: No logon possible (no hw ID received by mssg server).
In the SAP MMC, the message server (msg_server.exe) shows status "stopped".
The dev_ms file reports these errors:
[Thr 2548] *** ERROR => MsCommInit: NiBufListen(sapmsTST) (rc=NIESERV_UNKNOWN) [msxxserv.c 8163]
[Thr 2548] *** ERROR => MsSInit: MsSCommInit [msxxserv.c 1561]
[Thr 2548] *** ERROR => main: MsSInit [msxxserv.c 5023]
[Thr 2548] ***LOG Q02=> MsSHalt, MSStop (Msg Server 2900) [msxxserv.c 5078]
Problem Analysis
Work processes were able to start but the message server was not. The reason is because the "services" file is missing the SAP System Message Port entry. Example: SAPmsTST 3600/tcp
Solution
Edit the "services" file and add the entry. Then, re-start the instance. Make sure you specify the appropriate TCP port (e.g. 3600) for the message server.

5: The message server starts but the dispatcher doesn't

The dispatcher shows status "stopped" in the SAP MMC.
The "dev_disp" file shows these errors:
***LOG Q0A=> NiIServToNo, service_unknown (sapdp00) [nixxi.c 2580]
*** ERROR => DpCommInit: NiDgBind [dpxxdisp.c 7326]
*** DP_FATAL_ERROR => DpSapEnvInit: DpCommInit
*** DISPATCHER EMERGENCY SHUTDOWN ***
Problem Analysis
The keyword in the messages above is "service unknown" followed by the entry name "sapdp00". The dispatcher entry "sapdp00" is missing in the "services" file. Example: sapdp00 3200/tcp
Solution
Add the necessary entry in the "services" file. Example: sapdp00 3200/tcp Then, re-start the instance.

6: Work processes die soon after they start

All work processes die right after the instance is started.
The SAP MMC shows work processes with status "ended".
Only one work process shows status "wait".
An ABAP dump saying "PXA_NO_SHARED_MEMORY" is generated as soon as a user logs in.
The SAP MMC Syslog shows the following error multiple times: "SAP-Basis System: Shared Memory for PXA buffer not available".
Problem Analysis
The instance profile contains misconfigured memory-related parameters. Most likely the "abap/buffersize" instance profile parameter is set to high.
Solution
Edit the instance system profile at the OS level under /usr/sap//sys/profile and lower the value assigned to "abap/buffersize". Then, restart the instance. Also, it's important to find out if any other memory parameter were changed. If not, the system should start once the adequate memory allocation has been set to the the "abap/buffersize" parameter.


Tuesday, April 28, 2020

SAP HANA System Down, HANA not starting


how to perform checks if the SAP HANA instance is not starting. At the end of this guide, there will be frequently asked questions and common problems that are encountered.

Checks to perform

The first thing that is needed to be determined, is if the SAP HANA database is running. To do this run:

ps -ef | grep hdb



If the HANA database is running the following processes will be present

hdbnameserver

hdbpreprocessor

hdbcompileserver

hdbindexserver

hdbstatisticsserver (this may not be present as of post SP7 this could be merged into the Indexserver)



Please ensure that processes are being ran by the correct <SID>adm user incase they have multiple HANA's running on the system

If you see the running processes then please review the System Hang section.



To see if the HANA database will start try via putty going to /usr/sap/<SID>/HDB<instance#> and running

HDB start



If this fails go to /usr/sap/<SID>/HDB<instance#>/exe here you can try and run the processes manually

usually you will only need to call ./hdbnameserver and then the ./hdbindexserver and continue with the

rest if it is successful, if it is successful the issue could be with hdbdaemon or sapstartsvr and you

will check the associated logs.





Check the HANA trace files in the following location /usr/sap/<SID>/HDB<instance#>/<server name>/trace

or create a full system dump by following SAP Note 1732157 - Collecting diagnosis information for SAP HANA



The order of checking the trace files should be first the daemon, nameserver, indexserver, compileserver, and

preprocessor (statistics server would not cause the system to stop starting).

However, after checking the indexserver, you should be able to see where the error lies



Issues and Reported Problems

Common issues that we can see are:

Disk Full Error

In this the trace file will contain the words 'rc=24 no space left on device errors' for this please

review SAP Note 2083715 - Analyzing log volume full situations



Corrupt Log Segments

The trace file will say something like cannot find or cannot read a log segment at a hexadecimal address, the only

resolution to a corrupt log segment is to do a recovery that does not involve that log segment



Missing Log Segments

In the trace we will see 'Cannot open file "/<path_to_missing_logsegment>/logsegment_000_XXXXXXXX.dat", rc=2: No such file or directory'

for this please review SAP Note 1788692 - Index Server crash due to missing LogSegment file



Authorization Issues

In the trace we will see the message 'not authorized' in the trace, in this scenario check as the <SID>adm

user and see if that user can make a file in the location specified in the trace to verify this. If you

cannot create the file run the chmod command on the folder to allow reading and writing (ie chmod 764)



Hardware issue

There is no generic line in the trace would point to hardware, but if the issue is OS related or a disk cannot mount please follow the

hardware portion of the survival guide



HANA Not Starting after a failed hdblcm rename (hdbrename)

When you try to start HANA it fails with "process hdbdaemon HDB Daemon not running". No daemon, nameserver, or indexserver trace is created which indicates that it hasn't even gotten to the point of trying to start the services.

SAP Note 2142432 - SAP HANA does not start after a failed attempt to rename the HANA SID







System Crash

An SAP incident will have to be made with a full system dump (SAP Note 1732157 - Collecting diagnosis information for SAP HANA)

HANA up but SAP system not starting:

Check if a connection is possible to the database by running

R3trans -d

this will end with a return code. RC <8 is a successful connection to the database but rc=12 would be a failure. 

Check the trans.log which is produced to see further details about why the abap side of the SAP system could not connect to the database.

Here are some examples of common issues when R3trans d results in r=12



Your HANA DB rev is SPS9 (rev 90 or higher) and you see something similar to what is listed below:
"

4 ETW000  [     dev trc,00000]  Database release is HDB 1.00.090.00.1413897729                            54  0.055046 4 ETW000  [dbhdbsql.cpp,00000]  *** ERROR => Using non supported HANA version: 1.00.090.00.1413897729 4 ETW000  [dbhdbsql.cpp,00000]  *** ERROR => Min. version for this release must be 1.00.62

"

 Please see SAP Note 1952701  - DBSL supports new SAP HANA SP9 version number





Timezone and DST issues:

The system may come up but have dumps of ZDATE_LARGE_TIME_DIFF

Follow the guidelines at: http://scn.sap.com/docs/DOC-58741  

SAP Note 1932132 - SAP HANA : Large time difference between application server and HANA database

SAP KBA  2137138 - Timezone name incorrect after DST switch

Related Documents

 For DST preparations: http://scn.sap.com/docs/DOC-58741







Related Videos:





Related SAP Notes/KBAs

SAP Note 1732157 - Collecting diagnosis information for SAP HANA

SAP Note 2083715 - Analyzing log volume full situations

SAP Note 1788692 - Index Server crash due to missing LogSegment file






If SAP instance is not getting start/up

How to check, If SAP instance is not getting start/up in Linux – disp+work dispatcher IGS Watchdog Gateway ICM



How to check, If SAP instance is not getting start/up in Linux – disp+work dispatcher IGS Watchdog Gateway ICM

we can check & analyze, if an sap instance is not getting start.

There are many root causes for that. That may be

Sap buffer memory allocation issue.

Shared memory  allocation issue.

Dispatcher work process is in struct/hang state

May be port issue, etc.

Root causes and analysis – SAP instance:

Memory Allocation issues :

If your maintaining the system sizing & files system properly as per standard sap guides & as per business process requirement. After that system will allocate some types buffer memories as default min values. But some times, as per system installation working process, the work processes had some more additional requires the shared segments or increment/decrements of abap buffer sizes.In this case, we can check the analysis by executing below sappfpar command with <SID>adm user at OS level.


>/usr/sap/<SID>/SYS/exe/run/sappfpar check pf=/usr/sap/<SID>/SYS/exe/run/<profile_name> nr=<instance nuber> name=<SID> | more
Here, profile name should like “<SID>_D<instance no>_Hostname“.
After executing the command, you will get the all buffer memory allocation report & requirement with errors & warnings. As per requirement change the profile parameters values & confirm by re-executing the same.

Dispatcher is stopped :

Most of the time sap instance is not getting boot because of the respective dispatcher is not in running state. We can easily check & confirm with the below command.
> sapcontrol -nr <instance number> -function GetProcessList



Description: instance

In cause, if suppose the network issue has occurred, then respective all services will down in the server. Then if you try to start the instance services manually, It could not be start & the dispatcher is in stopped state with Gray rather than GREEN & running status. Solution : 

Find out the stopped work processes id’s (pid) by executing above command once again. Then kill that all work process manually.
> kill -9 <WP ID>


Then start the sap_instance again and also check the dispatcher status.

Gateway/Dispatcher Ports issue :

Some times both instances ASCS, PASS are started but respective Dispatcher is not in running status. Because while booting, the respective gateway/dispatcher ports 33<nn>, 32<nn> are not in free with in the server. Those ports are already established in that server. So, you need find out & kill them manually by using below commands.
fuser port/tcp or >netstat -nap | grep 33/32  : to find listening ports>fuser -k port/tcp  : to kill the listening port.
Otherwise simple reboot the application server.


Once the Database is up and running, then it should be connect through the <SID>adm user from Application server. You can cross verify it by using below command. Here, R3trans should be finished with ‘0000’.
#sidadm> R3trans -d


Description: instance
If not, it may cause due to the dispatcher & gateway not working, you can cross verify from Step 2 again. You can also check the trans.log as like below,
>su – <sid>adm
>cat trans.log



Buffer instance IPC cleanup process :

You can cleanup the ipc buffer by executing the below commands at instance level.

Switch to <sid>adm user then run the below command

cleanipc <instance no> remove
OR

cleanipc all remove

Note : Still if you face any issue, please check the below log files, which are exist under the instance work directory. Take the action accordingly.
dev_disp
dev_icm
dev_rd
dev_w0, dev_w1 


Tuesday, March 31, 2020

Log and Traces -Transactions


SYSTEM LOG(SM21)
a) Can be used to detect and correct errors in our SAP system and its Environment.
b) SAP application servers record events and problem in system logs.
c) Every SAP application Server has a local log that contains the messages output by this server.
Dump Analysis(ST22)
a) If unpredictable errors occurs during run-time when you call an ABAP program , a run-time error that generates a short dump can occurs.
b) By default, short dump are stored in the system for 14 days.
c) You can delete short dumps in accordance with a time specification using the reorganize function ,which you call by choosing Goto -->Reorganize .
d) one can save a short dump without a time limit using the KEEP function ,which one can choose from detail view under short Dump -->KEEP/RELEASE.
Characteristics of Dump Analysis:-
1) If a run time error occurs ,a short dump is generated.You can use transaction ST22 to analyzer this short dump.
2) Dump data is stored in the database.
3) Dump data can be reorganized.
4) Individual short dump can be flagged for retention.
System Trace (ST01)
To record the internal SAP activities ,such as authorization checks,database access ,kernel functions and RFC calls.
The system trace is used for analyzing:
a) Authorization checks
b) Kernel functions
c) Kernel modules
d) DB accesses (SQL Trace)
e) Accesses to table buffers
f) Lock operations (client-side).
PERFORMANCE TRACE(ST05)
The Performance trace is used for analyzing:
a) Database calls
b) Lock management calls
c) Accesses to table buffers
d) Remote calls of reports and transactions
e) Individual trace records
f) SQL statements.
DEVELOPER TRACE(ST11)
Developer traces are recordings that contain technical information and that are used if errors occur.
a) Can be read by using Transaction AL11.
b) Browse to directory usr/sap//d*/work.
c) Developer trace can be viwed at dev_* files.
d) Can be accessed in Transaction: SM50. Process → Trace → Display File.

Monday, March 30, 2020

(Un)lock the shadow instance/Starting and stopping the shadow instance manually

Symptom

When trying to logon to the shadow instance with a user other then DDIC ( in one of the MAIN_SHDPREPUT*, MAIN_SHDRUN/* , MAIN_SHDIMP/SUBMOD_SHD2_RUN/* phases)
 you get the message "Upgrade in progress: no login is possible".
In this case the shadow instance needs to be unlocked.

Solution

To unlock the shadow system, enter the following commands

UNIX:
cd /<update directory>/abap/bin   (e.g. /usr/sap/PRD/SUM/abap/bin)
./SAPup unlockshd
WINDOWS:
<drive>
cd \<update directory>\abap\bin   (e.g. \usr\sap\PRD\SUM\abap\bin)         
.\SAPup unlockshd
REMARK:
If the shadow instance is not working in the current upgrade phase, one of the following message will be displayed:
At this time of the procedure, the shadow instance does not yet exist since you are before phase START_SHDI_*!
or
At this time of the procedure, the shadow instance does not exist anymore since you are passed phase STOP_SHDI_*!"
Conclusion: Depending on the update phase another instance needs to be (un)locked.
Please follow the guide
Lock/unlock the standard instance during update.

Log on to the shadow system and correct the error.

To lock the  shadow system again, enter the following command:

UNIX:
./SAPup lockshd
WINDOWS:
.\SAPup lockshd

Related SAP notes/KBAs

SAP KBA 1901463 - How to unlock the SAP system to perform correction(s) during an upgrade.


Starting and stopping the shadow instance manually

 

Symptom

The shadow instance start/stop is controlled by the upgrade.
 Attention! Start/stop the shadow instance manually only, if it is really requested, unwanted start/stop can impact the whole update.
Possible situations, when the shadow instance needs to be started/stopped manually:
REMARK: Some instance parameter can be activated dinamically without instance restart, check the parameter documentation in the transaction RZ11.
  • Restart the shadow instance in the nZDM scenario  

Solution

To stop the shadow instance manually, proceed as follows:

Step 1

UNIX:           cd <update directory>/abap/bin           example:     cd  /usr/sap/PRD/SUM/abap/bin    
WINDOWS: <drive>                                                 example:     D:                     
                    cd <update directory>\abap\bin           example:     cd \usr\sap\SUM\abap\bin

Step 2

UNIX:            ./SAPup stopshd    
WINDOWS:  .\SAPup stopshd

Step 3

The following output can be displayed:
Enter the path of your root directory [D:\usr\sap\SUM\abap]: Root directory = 
Confirm if the correct SUM directory is displayed and press Enter
The following output will be displayed:
Stopping shadow system ...
Stopping service ...

To start the shadow instance manually, proceed as follows:

Step 1

UNIX:           cd <update directory>/abap/bin           example:     cd  /usr/sap/PRD/SUM/abap/bin    
WINDOWS: <drive>                                                 example:     D:                                            
                     cd <update directory>\abap\bin           example:     cd \usr\sap\SUM\abap\bin

Step 2

UNIX:            ./SAPup startshd    
WINDOWS:  .\SAPup startshd

Step 3

The following output will be displayed:
 Starting shadow system ...
Starting service ...
waiting 4 seconds for system to come up ...
testing if system is available ...
REMARK: If the shadow instance is not working in the current upgrade phase, one of the following message will be displayed:
  At this time of the procedure, the shadow instance does not yet exist since you are before phase START_SHDI_*!
or
At this time of the procedure, the shadow instance does not exist anymore since you are passed phase STOP_SHDI_*!"
Conclusion: Depending on the update phase another instance needs to be started or stopped.
To identify the currently used instance or to analyse other start/stop errors please follow the troubleshooting guide
 Troubleshooting instance start-stop situations during update

Caution when using IBM i (AS/400, iSeries, i5/os)!
Before the actual command for starting and stopping the shadow instance for the platform IBM i, you must start a QP2TERM session and supplement the execution environment, for example, in the following way:
1. Perform the following steps as <SID>OFR or as <SID>ADM (depending on the user concept that is active).
2. Call QP2TERM. Then in the QP2TERM session:
3. cd <upgrade_directory>/bin
4. export LIBPATH=.
5. Execute the command R3up/SAPup/SAPehpi specified below.
6. You can exit the QP2TERM session using F12/F3.
  • Starting the shadow instance
           Start SAPup   as the user <sid>adm as follows:
              SAPup  startshd
  • Stopping the shadow instance
           Stop SAPup   as the user <sid>adm as follows:
             SAPup   stopshd



Troubleshooting shadow instance start problems  

If the instance terminates with error during startup, troubleshooting steps can be found on the wiki page
Troubleshooting instance start-stop situations during updateChapter  Typical upgrade phases, where the shadow instance is used:     

Related SAP notes/KBAs

SAP note 398100 - Shadow instance during upgrade
SAP note 525677 - Problems when starting shadow instance 


Troubleshooting instance start-stop situations during update

Purpose

During update we have various situations when an instance will be started/stopped by the update tool and troubleshooting is needed.
Follow this guide to find symptoms and solutions.

Solution

1. Identify the upgrade phase

It is shown on the SUM GUI or can be found at the end of the upgrade log file
UNIX: <update directory>/log/SAPupConsole.log or <update directory>/log/SAPup.log
WINDOWS: <update directory>\log\SAPupConsole.log or <update directory>\log\SAPup.log

2. Select the upgrade phase where the start/stop situation should be checked:

Typical upgrade phases, where the shadow instance is used:
START_SHDI_PREPUT
START_SHDI_FIRST
ACT_UPG
START_SHDI_SHD2
STOP_SHDI_RES
Typical upgrade phases where the standard instance is used:
STARTSAP_TRANS
STOPSAP_TRANS
STARTSAP_NBAS
STARTSAP_PUPG or STARTSAP_TBUPG

3. If the instance needs to be started or stopped manually

please follow the steps on the pages
Starting and stopping the shadow instance manually
Starting and stopping the standard instance during update manually


Start/stop the instance manually only, if it is really requested, unwanted start/stop can impact the whole update.

Related documents

SAP Startup Troubleshooting Guide for Netweaver Application Server    


Sunday, March 29, 2020

SAP does not start

SAP does not start

  • SAP system does not start.
We have received many incident reports (aka message) whose titles are “SAP system does not start”.
For SAP support engineers, this title has broad meanings. Since SAP system is based on the client – server technology and consists of many processes or threads, there are many places to be checked in order to understand what the exact problem you are facing.
What do you mean by “SAP system does not start”?
By reading the description of the incident report further, the reporters describe the situation like: “.. the users cannot logon after restart..”, “.. the SUM tool says startup failure..”, “.. my message server didn’t come up..”, “..I cannot see OS processes..”, “..I started my own startup scripts but nothing happened..”, and so forth.
Some of those descriptions are still quite vague, and some are quite to the point of the root cause.
First of all, please check this question:
Are Message Server, a dispatcher and work processes up and running?
Yes: -> this means that the startup of SAP system was finished successfully. It can be said that the problem is the logon itself, and not SAP system startup. If this is the case, then please avoid using the title: “SAP system does not start”.
No: -> literally it means that SAP system DID NOT startup. Its startup sequence had some kind of problems.
Let’s have a look this startup problem further.
This wiki aims to offer troubleshooting guides for startup problems. With this guide, we hope you can analyze and solve the problem by yourself. In some cases, you may still need to raise an incident ticket (aka message). Then, we hope you can narrow down the problem and provide us detailed information so that we can instantly assist you.
Startup sequence
897933 - Start and stop sequence for SAP systems
In short, SAP system instances can be started up by the following sequence:
  1. Database instance
  2. Central instance or (A)SCS
  3. Application servers (Dialog Instances)
*Database startup problem is not discussed in this wiki.
The following link explains standard methods to start SAP systems:
http://help.sap.com/saphelp_nw70ehp1/helpdata/en/47/fd7230eca159e8e10000000a421937/frameset.htm
SAP MC, SAPMMC and startsap script needs “SAP start service”:
http://help.sap.com/saphelp_nw70ehp2/helpdata/en/b3/903925c34a45e28a2861b59c3c5623/frameset.htm
As you can see from above link, many processes are involved in the startup. Among those processes, the sapstartsrv process plays an important role of controlling a SAP instance.
The types of the processes which need to be executed in the system are declared in the instance profile (or start profile). Then, the sapstartsrv process starts up the specified processes.
During this startup sequence, each process generates its trace file on DIR_HOME directory.
Common problems and solutions
Quite common problems are related to the sapstartsrv process, especially following two symptoms: “The process doesn’t exist” and “The process exists but does not respond”.
“The process doesn’t exist”
On UNIX/LINUX, use an OS command ‘ps’ to check the process:
ps –ef | grep sapstartsrv
When the process exists, the above command shows like this:
  abcadm  7012358        1   0 17:39:33      -
0:05 /usr/sap/ABC/DVEBMGS00/exe/sapstartsrv pf=/usr/sap/ABC/SYS/profile/START_DVEBMGS00_walldorf -D -u abcadm
On WINDOS, use WINDOWS’ task manager tool to see if the service is running or not.
                     

If the sapstartsrv process doesn’t exist, please consult following SAP notes and start up.
1762827 - Startup of Instance Service failed in UNIX or Linux environment

“The sapstartsrv process exists but does not respond”
The OS command ‘PS’ or the taskmanager shows the process. But the sapstartsrv doesn’t respond to any actions. Then, most likey, the UNIX domain socket (/tmp/.sapstream<port-no>) or Windows named pipe (\\<host>\pipe\sapcontro_<xx>) may be broken. If it still exists, please manually delete. And start the sapstartsrv service again.
Other cases…
Case 1:
Found the following error information in the startsap.log.
exec(): 0509-036 Cannot load program disp+work because of the following
errors:
rtld: 0712-001 Symbol ixml_iXMLParser_IsNormalizingAtt was referenced
      from module disp+work(), but a runtime definition
      of the symbol was not found.
rtld: 0712-001 Symbol ixml_iXMLParser_SetNormalizingAtt was referenced
      from module disp+work(), but a runtime definition
Root cause:
The library path was not configured correctly after upgrade.
Solution:
Configure the library path.
SAP Note 1265456 SAP system does not start after kernel upgrade
SAP Note 1104735 Upgrade to the new instance-specific directory on UNIX
Case 2:
The sapstartsrv service is up but the dispatcher doesn’t startup.
sapcontrol -nr 00 -function GetProcessList
30.11.2013 18:46:21
GetProcessList
OK
name, description, dispstatus, textstatus, starttime, elapsedtime, pid
disp+work, Dispatcher, GRAY, Stopped, , , 26680
igswd_mt, IGS Watchdog, GREEN, Running, 2013 11 30 12:57:32, 5:48:49,
26681
Root cause:
The zombie process of the dispatcher existed.
Solution:
Kill the zombie process.
Case 3:
On NT, frequently failover in few minutes.
Root cause:
The version of the sapstartsrv.exe was different between DIR_CT_RUN and DIR_EXECUTABLE.
Solution:
Copy the sapstartsrv.exe from DIR_CT_RUN to DIR_EXECUTABLE.
SAP Note 1043592 MSCS: Cluster Resource Monitor Crashes on W2K3 SP2
SAP Note 1345206 Handling and preventing sapstartsrv.exe corruptions
SAP Note 1375494 SAP system does not start after applying SAP kernel patch
Case 4:
Startsap.log says:
(1159276) Starting: local /usr/sap/ABC/SYS/exe/run/igswd_mt -
mode=profile pf=/usr/sap/ABC/SYS/profile/ABC_DVEBMGS00_walldorf
(442404) Waiting for Child Processes to terminate.
(442404) **** 2010/04/10 08:07:11 Child 1155152 terminated with Status
255 . ****
(1155152) **** 2010/04/10 08:07:11 No RestartProgram command for
program 2  ****
The information above is very generic. stderr5 file shows more detailed information.
Command:  /usr/sap/ABC/SYS/exe/run/igswd_mt
           -mode=profile
           pf=/usr/sap/ABC/SYS/profile/ABC_DVEBMGS00_walldorf
(1159276) **** Failed to Start Local Command. No such file or directory
Root cause:
The necessary files weren't copied correctly.
Solution:
Copy the igswd_mt to /usr/sap/ABC/SYS/exe/run
Case 5:
After kernel patching, the system doesn't come up but the startsap commands runs successful. The dispatcher was immediately down and no
work process is active.
Stderr2 says:
(1114238) New Child Process created.
(1114238) Starting local Command:
 Command:  dw.sapABC_DVEBMGS01
           pf=/usr/sap/ABC/SYS/profile/ABC_DVEBMGS01_walldorf

Fri Mar  5 08:59:56 2010
Profile configuration error detected, use temporary corrected setup
Shared Pool 40: ipc/shm_psize_40 = 2180000000 (too small)
Shared Pool 40: (smaller than min requirement 2186246040)
Shared Pool 40: (estimated size assumed 2190000000)
*** ERROR => Illegal ShmAdm Slot. Key=2104967296.
*** ERROR => ShmGet: Inv PoolKey=2104967296, Key=40, Flag=1, Ptr=1152921504606843304.
***
Root cause:
Shared memory area may need to be adjusted for new environment (i.e, after patching).
Solution:
Run sappfpar check pf=<profile> and adjust ipc/shm_psize_xx to suggested values.
Case 6:
The dispatcher went down right after startup. The dev_disp shows:
rdisp/http_min_wait_dia_wp : 1 -> 1
***LOG CPS=> DpLoopInit, ICU ( 3.4 3.4 4.1) [dpxxdisp.c   1706]
***LOG Q0K=> DpMsAttach, mscon ( CLMAGMACI) [dpxxdisp.c   12674]
*** DP_FATAL_ERROR => DpMsAttach: local hostname 'SRMAGMA98' is resolved
 to loopback address (cf. SAP note 1054467 for details)

*** DISPATCHER EMERGENCY SHUTDOWN ***

Root Cause
Incorrect setup of the loopback address.

Solution:
#1054467 Local host name refers to loopback address

Case 7:
The dispatcher went down right after startup. The dev_disp shows:
*** ERROR => gateway (pid 29622344) died [dpxxdisp.c   16763]
*** DP_FATAL_ERROR => Gateway died with status 2 - I better exit now

*** DISPATCHER EMERGENCY SHUTDOWN ***

The gateway trace dev_rd shows:
***LOG Q0I=> NiIBindSocket: bind (67: Address already in use)
[nixxi.cpp 3740]
*** ERROR => NiIBindSocket: SiBind failed for hdl 9/sock 12
    (SI_EPORT_INUSE/67; I4; ST; 0.0.0.0:3300) [nixxi.cpp    3740]

***LOG S0V=> GwStopGateway, gateway stopped () [gwxxrd.c     13096]

Root Cause:
The service port 3300 was declared twice in the /etc/services file.

sapgw00         3300/tcp
sapgw00s        3300/tcp

Solutoin:
Make sure only one service is available.
sapgw00         3300/tcp
#sapgw00s        3300/tcp

Case 8:
The dispatcher went down right after startup. The dev_disp shows:
sapstart.log
(11925) Starting local Command:
 Command:  dw.sapPIP_DVEBMGS00
           pf=/usr/sap/PIP/SYS/profile/PIP_DVEBMGS00_unjspfsappip

(11856) Waiting for Child Processes to terminate.
(11856) **** 2013/09/19 10:31:13 Child 11925 terminated with Status 0 . ****

dev_disp
*** ERROR => NiDgHdlBindName: invalid hostname 'localhost' (rc=-2;hdl 1) [nixx.c       4341]
*** ERROR => DpCommInit: NiDgHdlBindName failed: -8 [dpxxdisp.c   10339]
*** DP_FATAL_ERROR => DpSapEnvInit: DpCommInit
*** DISPATCHER EMERGENCY SHUTDOWN ***

Root Cause:
“hosts” file was modified and “localhost” was misconfigured.

Solution:
Configure “localhost” in the “hosts” file correctly.
Case 9:
The sapstartsrv is up.
When you execute sapcontrol without host opetion, you get: NIECON_REFUSED (WSAECONNREFUSED) error. However, it success with -host opetion.
For example:
sapcontrol -nr 00 -function GetVersionInfo
FAIL: NIECONN_REFUSED (WSAECONNREFUSED: Connection refused),
NiRawConnect failed in plugin_fopen()
but
sapcontrol -nr 00 -host saphost.sap.co:50013 -function GetVersionInfo -> works!
Root Cause:
Please refer SAP Note 2062508 sapcontrol commands fail with error "NIECONN_REFUSED (WSAECONNREFUSED: Connection refused)" during upgrade
Solution:
Comment out parameter service/hostname for the SAP start service and restart sapstartsrv.


The sapstartsrv exisits. -function GetProcessList doesn't show programs like dispatcher, gateway, etc.
In normal case, you will see something like this:
----
sapcontrol -nr 00 -function GetProcessList
GetProcessList
OK
name, description, dispstatus, textstatus, starttime, elapsedtime, pid
msg_server, MessageServer, GREEN, Running, 2014 02 25 05:45:24, 0:00:53, 7326
disp+work, Dispatcher, GREEN, Running, Message Server connection ok, Dialog Queue time: 0.00 sec, 2014 02 25 05:45:24, 0:00:53, 7327
igswd_mt, IGS Watchdog, GREEN, Running, 2014 02 25 05:45:24, 0:00:53, 7328 
---
-->here, you can see programs (message server, disp+work and IGS) are up and running.
But in this problematic situation, something like....
----
sapcontrol -nr 00 -function GetProcessList
GetProcessList
OK
name, description, dispstatus, textstatus, starttime, elapsedtime, pid
----
--> No programs are listed .
This is because 'Start_Program_n' command with the program is missing in the profile file which sapstartsrv starts with.
For this 'Start_Programm_n", please refer the following link:
http://help.sap.com/saphelp_nw74/helpdata/en/9c/ac0d50a40af26be10000000a423f68/frameset.htm
(NOTE!!! since kernel 74x release, 'Start Profile' is obsolated. You cannot use 'Start Profile' like ealier release. All the information in Start Profile can be marged to Instance Profile)

After upgrading to 74x system, SAP doesn't startup.
In most cases, the reason is the sapstartsrv is pointing to Start Profile. Start Profile is obsolated since 74x kernel. You need to recofigure your profile.
  • Delete Start Profile physically from 'profile' dir.
  • In UNIX, adjust /usr/sap/sapservices file.
  • In WINDOWS, adjust the SAP service.
  • Delete UNIX domain socket or Window's named pipe, and restart sapstartsrv.

UNIX Domain Socket .sapstream5xx13 was not enabled...
Case 1:
sapstartsrv process exists but the corresponding UNIX Domain Socket doesn't exist.
Checki sapstartsrv.log. It ends at CCMS agent initialization phase:
cat sapstartsrv.old
---------------------------------------------------
trc file: "sapstartsrv.log", trc level: 0, release: "741"
---------------------------------------------------
pid 2134
[Thr 01] Mon Sep 15 10:09:13 2014
HistoryLog_Init: logfile (/usr/sap/ABC/SCS00/work/history.glf) already exists, check parameter
No halib defined => HA support disabled
CCMS agent initialization for instance type SCS: return code 0.
-->that's all.... sapstartsrv should initialize sapcontrol webservice....
Normally, you should see something like:
CCMS agent initialization for instance type SCS: return code 0.
CCMS agent disabled by profile parameter ccms/enable_agent = -1.
Initializing SAPControl Webservice
Auto PSE update thread started
Starting AutoRestart thread
Starting WebService SSL thread
AutoRestart thread started
Starting WebService thread
Webservice thread started, listening on port 55013
Webservice SSL thread started, listening on port 55014
Webservice SSL thread using default SAP SSL credential
Trusted https connect via Unix domain socket '/tmp/.sapstream55014' enabled.
Trusted http connect via Unix domain socket '/tmp/.sapstream55013' enabled.
Root Cause:
CCMS agent has been running already by saphostcontrol.
Solution:
set ccms/enable_agent = 0 in the instance profile.
Case 2:
A trace file called sapstartsrv_ccms.log was generated in work dir. The content of i ends with the information:
INFO: Checking lock file.....
Root Cause:
Please refer SAP note 1916333 sapcontrol error "FAIL: NIECONN_REFUSED" due to agent.log file
Solution:
set ccms/enable_agent = 0 in the instance profile.