Concurrent Manager

Concurrent Mangers log files are located in the $APPLCSF/$APPLLOG location.

Where is Concurrent Manager log file location.

By default standard location is $APPLCSF/$APPLLOG , in some cases it can go to $FND_TOP/log as well

cd $APPLCSF/$APPLLOG

For ICM Log                                       –> ls -lrt *$TWO_TASK*

For Standard manager Log                  –> ls -lrt w*.mgr

For Conflict Resolution manager Log –> ls -lrt c*.mgr

Question 1: What are different types of concurrent manager?

Answer There are several types of concurrent manager. Important ones:

Internal Manager

Standard Manager.

Conflict Resolution Manager

Internal monitors

Service Manager

Transaction Manager

Custom Manager Defined

Read more on below link

Oracle Concurrent Manager

Question 2: What is Internal Concurrent Manager?

Answer : It is responsible for controlling all other concurrent managers. Its main task is to make sure that all other concurrent managers are up and running. It controls other manager through the requests made to the service manager. It also starts, stops and restarts the service manager for all nodes.

Question 3: What is Conflict Resolution Manager (CRM)?

Answer: It takes care of resolving the program incompatibilities and checks if a request in queue can be run in parallel with the running request. If a program is identified as run alone, then it prevents concurrent managers from starting other programs in the same conflict domain.

Question 4: What is Standard Manager?

Answer : Standard Manager is the master concurrent manager. It is always running and can take care of processing any concurrent request. If at all, no other manager is assigned to a program, that program will be picked by standard manager.

Question 5:

What happened behind the scene when a concurrent request is submitted?

Answer

1) Once a concurrent request is submitted by the user, the table FND_CONCURRENT_REQUESTS is automatically updated with the details of the request. The table is also updated with the information about the schedule of the concurrent request whether it’s immediately scheduled or scheduled at a fixed time.

2)a) If the request is incompatible/constraints defined ,Once the request time to run is arrived,it status is set to pending/Standby.Now the conflict resolution manager takes care of the request and finds out what are the incompatibilities and set the status pending normal when the incompatibilities are cleared.

b) If there are no incompatibilities then Once the request time to run is arrived,it status is set to pending/Normal

c) ALL the standard concurrent Manager and special manager continuously poll the FND_CONCURRENT_REQUESTS table .The job of a concurrent manager is to execute concurrent requests that are in Pending / Normal phase / status and that it is qualified to run according to its specialization rules.

Concurrent Manager Processes

– Act independently

– Select only requests that: (a) match the manager specialization rules, (b) are Pending/Normal, (c) have a requested start time <= sysdate

d) Once the request is processed, the FND_CONCURRENT_REQUESTS table is updated with the status.

Question 6: Business user create the task that Concurrent request is taking a lot of time to complete. What will be your approach for debugging it?

Answer

1) First find the status of the concurrent request. It may be scheduled later on or it could be pending/standby mode or all the concurrent manager are occupied running other request. If it is pending/standy, we need to find the incompatible program running and inform user.Many times the users schedule the request to run at a later time.

2) Find out the database sid of the concurrent request and check it is waiting on any locks. We will kill the blocking session to order to complete the job

3) We can run a trace on the request id to find the sql running and then generate the explain plan for it. You can see if the sid is stuck on particular sql . If it is particular sql, then it is good to check the statistics of the table involved . We can look for tuning option for that query

4) We can check the parameters with which the request is run. (For example, once a user came saying the request is not printing the output. On Checking the possible things, it was realized that he scheduled the request with print copies = 0.)

Question 7: What happens when the internal concurrent manager dies abruptly? Are all the managers also killed immediately after it?

Answer

No ALL the standard manager keep running and executing the request. if the internal manager dies, below queue control request are not performed

a) Starts all other processes.

b) Executes “control requests” submitted by the administrator.

c) Activate/Deactivate/Abort Concurrent Manager

d) Terminate Concurrent Request

e) Monitors processes, restarting any that failed.

f) Sets the target number of processes for each service based on the current work shift.

Question 8: Does the internal manager run or schedule any request for itself?

Answer

No, the internal manager does not run or schedule any requests. It has nothing to do with scheduling requests, or deciding which manager will run a particular request. Its function is only to run ‘queue control’ requests

a) Starts all other processes.

b) Executes “control requests” submitted by the administrator.

c) Activate/Deactivate/Abort Concurrent Manager

d) Terminate Concurrent Request

e) Monitors processes, restarting any that failed.

f) Sets the target number of processes for each service based on the current work shift.

Question 9: How do I process more concurrent requests in parallel?

Answer

We can increase the target processes of the concurrent manager in order to increase the parallelism. This can be done using define concurrent manager form or through direct update from sqlplus

Question 10 : If the internal manager goes down, do I need to kill all the managers before restarting the internal manager?

Answer

No, if the internal manager goes down you need not kill all the managers. You can simply start the internal manager using startmgr.

Concurrent Manager troubleshooting

Question 11 : What are the problems u have faced while shutting down applications?

Answer

While shutting down application generally concurrent manager won’t go down because some or the other request may be running. We will see what are the concurrent requests running by querying fnd_concurrent_requests, fnd_concurrent_program_vl, v$session, v$process and v$sqltext.

If that request is only doing some select statement then we will kill those requests, otherwise we will check what time it will take to complete by querying the previous runs of that request and then we will decide what to do.

Oracle Concurrent Manager Secrets

Question 12: What are Internal Monitors ?

Answer: Internal Monitors are used specifically in PCP to allow for ICM failover to other available middle tier nodes.

a) Place an Internal Monitor on any node where the ICM can start in case of a failure.

b) Internal Monitors are seeded on every registered node by default.

c) If the ICM goes down, the Internal Monitor will attempt to start a new ICM on the local node.

d) If multiple ICMs are started, only the first will stay active. The others will gracefully exit.

Question 13: Can I delete concurrent manager?

Answer:

Yes, you can delete any concurrent manager. For deleting, query for the manager in the defined concurrent manager form and then delete the row.

Deleting the predefined concurrent managers is not recommended and it should never be done. Deletion may cause instability in the system.

Question 14: How can you know which trace file is created for the particular request?

Answer

You can find out the same using the script given below. The trace will be located in the udump location of the database server.

prompt

accept request prompt ‘Please enter the concurrent request id for the appropriate concurrent program:’

prompt

column traceid format a8

column tracename format a80

column user_concurrent_program_name format a40

column execname format a15

column enable_trace format a12

set lines 80

set pages 22

set head off

SELECT ‘Request id: ‘||request_id, ‘Trace id: ‘||oracle_Process_id, ‘Trace flag: ‘||req.enable_trace, ‘Trace Name: ‘||dest.value||’ ‘||lower(dbnm.value)||’ora’||oracle_process_id||’.trc’, ‘Prog. Name: ‘||prog.user_concurrent_program_name, ‘File name: ‘||execname.execution_file_name||execname.subroutine_name , ‘Status :’||decode(phase_code, ‘R’, ‘Running’)||’ ‘||’-‘||decode(status_code, ‘R’, ‘Normal’), “SID Serial: “||ses.sid||” , “||ses.serial#, “Module : “||ses.module

from fnd_concurrent_requests req,

v$session ses, v$process proc,

v$parameter dest, v$parameter dbnm,

fnd_concurrent_programs_v1 prog,

fnd_executables execname

where req.request_id = &request

and req.oracle_process_id=proc.spid(+)

and proc.addr = ses.paddr(+)

and dest.name=’user_dump_dest’

and dbnm.name=’db_name’

and req.concurrent_program_id =

prog.concurrent_program_id

and req.program_application_id =

prog.application_id

and prog.application_id =

execname.application_id

and

prog.executable_id=execname.executable_id;

Top 30 Most Useful Concurrent Manager Queries

Question 15: Explain how parallel concurrent processing(PCP) works?

Answer

In case of parallel concurrent processing, all the managers are assigned a primary and a secondary node. The managers are started in their primary node by default. In case of node failure or Oracle instance failure, all the concurrent managers on that node are switched to their secondary nodes. Once the primary node is available again the concurrent managers on the secondary nodes are migrated back to the primary node. During the migration process, a manager may be spread across both primary and secondary nodes.

In case of parallel concurrent processing, it may happen that in a node where parallel concurrent processing is configured, the Oracle instance may or may not be running. The node which is not running Oracle, the concurrent managers connects via Net8 to a node which is running Oracle.

The internal concurrent manager can run on any node, and can activate and deactivate concurrent managers on all nodes. Since the internal concurrent manager must be active at all times, it needs high fault tolerance. To provide this fault tolerance, parallel concurrent processing uses internal monitor processes. The job of the internal monitor process is to constantly monitor the internal manager and start it when it fails. Only one internal monitor process can be active on a single node. You decide which nodes have an internal monitor process when you configure your system. You can also assign each internal monitor process a primary and a secondary node to ensure fail over protection. Internal monitor processes, like concurrent managers, can be assigned work shifts, and are activated and deactivated by the internal concurrent manager.

Parallel Concurrent Processing

Question 16: What are the circumstances in which you need to bounce the concurrent manager?

Answer: There can be many situation where you need to bounce the concurrent manager

a) When you modify the definition of the printers

b) When you modify the environment variables. Suppose you have changed the APPLTMP and APPLPTMP variable.

c) When all the requests are pending and hanging and no processing happening

d) patch application requires the bounce of the CM

e) We have many global hangs in the system due to locks by several concurrent manager and other processes

Question 17: What are the reasons a concurrent manager hangs?

Answer:

The concurrent manager hangs due to many reasons. A few of them are:

– Long running jobs

– The internal manager was activated by someone other then owner of the application system

– The operating system files system is full

– It’s not able to create the log file

– You’ve shut down the internal manager, but actual has a number in it

– The database is hanging may be because the archive log files have filled

– Pending/standby requests are too many

Question 18: How can we enable/disable Conflict Resolution Manager?

Answer: It can be done using profile options “Concurrent: Use ICM”. Set it to “Y” to enable Conflict Resolution manager. To disable it, set the profile option to “N”.

Question 19: What are Transaction Managers?

Answer : Transaction managers provide synchronous job processing by continually monitoring a DBMS pipe for requests to come through from a client-side application. The job of a transaction manager is to process this job immediately and send information back to the client using the pipe.

a) Transaction Managers Provide Synchronous Job Processing

b) A client makes a request for a specific transaction manager to run a program, and waits for the results of that program

c) Product teams’ programs are linked directly into the transaction manager executables

d) PO, CRP, INV, AR, and OE all ship transaction managers

Question 20: How the mechanism of viewing log and output files works from browser?

Answer

The sequence of events is as follows:

1. A user within an Applications session requests to view a log or out file.

2. The browser receives the request and spawns the cgi program FNDWRR.exe

3. FNDWRR.exe logs into the database and queries FND_CONCURRENT_REQUESTS to discover on which node the files for this request are stored.

4. FNDWRR.exe constructs the service name for the file server on that node. And makes the tns call to contact the listener for this service name.

5. The listener responds by spawning the local FNDFS executable, as defined in it’s listener.ora file. Now, FNDFS and FNDWRR.exe are able to communicate directly now, using RPC calls.

6. FNDWRR.exe requests FNDFS to transfer the file that was selected by the user.

7. FNDFS transfers the file contents to a temporary file directory on the web server’s node.

8. The web server displays the file contents to the user.

Question 21: Why does the Concurrent manager put a concurrent program into a queue? Why doesn’t the manager simply let the program run?

Answer: Because at any given point in time a concurrent manager can run no more than say 10 programs concurrently. This figure of 10 is configurable of course. First the manager puts a submitted program into a queue, next the manager checks if there is a slot available (i.e. Less than 10 programs are currently running). If a slot is found available, the concurrent manager then runs the program, or else it keeps the concurrent program in a queue with status Pending.

Question 22: What to do if a request is Inactive/No Manager

Answer: This is quite tricky questions

a) Make certain that there is at least one active manager with specialization rules that allow the program.

b) If you have confirmed the previous point, then the problem may be a stale Worker Request View

– The view is used internally to map requests to managers

– The view is regenerated when managers are created, or specialization rules are altered

c) You can manually regenerate the view

FNDLIBR FND FNDCPBWV apps/apps SYSADMIN ‘System Administrator’ SYSADMIN

Question 23: What are Service Managers

Answer: Service Managers are spawned on the middle-tier nodes of a GSM enabled system in order to act as an agent of the ICM. When the ICM sees that it needs An Service Manager to perform some function, such as start a concurrent manager process, on a middle-tier node, it will make remote procedure control (RPC) calls to the Apps listener on that node to start the Service manager. Once the Service manager has been started and initialized, the ICM communicates directly to the SM through RPC, giving it information to manage the services on that node. The SM is spawned from the APPS TNS Listener.The APPS TNS Listener must be started on every middle-tier node in the system, and started by the user that starts ICM (e.g. applmgr) .TNS Listener spawns Service Manager to run as agent of ICM for the local node

The Service Manager is started by ICM on demand when needed. If no management actions are needed on a node a Service Manager will not be started by ICM until necessary. When ICM exits its Service Managers exit as well.

Question 24: What all is performed by ICM Process Monitor?

Answer: The ICM itself and each process spawned by the ICM has an entry in FND_CONCURRENT_PROCESSES and holds a uniquely named DBMS lock.

The unique lock of the ICM has a format of FNDCPLK_ICM. This database session lock is the method that the ICM ensures each PMON cycle that manager and service processes are still alive. If the ICM can get the DBMS session lock of a process, the ICM will start a new process for that manager or service.