Oracle Enterprise Manager Agent Upload Troubleshooting

There are times when Oracle Enterprise Manger agent stop uploading.


There are many reasons for the agent not being able to upload.

Examine the agent

1. First I always check the status of the agent
 

emctl status agent

2. Examine status output, may see that the agent is not uploading by see
 

Last successful upload : (none)
Last attempted upload : (none)
Total Megabytes of XML files uploaded so far : 0.00
Number of XML files pending upload : 828
Size of XML files pending upload(MB) : 56.61
Available disk space on upload filesystem : 8.50%
Last successful heartbeat to OMS : 2012-07-21 08:45:39

3. Next I always attempt to do a manual upload for the agent to check/verify upload problem
 

emctl upload agent

4. Check the following log file for errors
 

$AGENT_HOME/sysman/log/emagent.trc

Other logfiles are in the $AGENT_HOME/sysman/log/ directory.

In Most cases it is due to a bad .xml file or unable to contact the management service. If it is due to contacting the management service you will need to fix the OMS Service or connectivity to the service, however if it is due to a bad xml file you can then try to just remove that file from the upload location. I usually save the file to another location so that I can send the file to Oracle Support where hopefully they can tell me why the file would not upload.Once you have removed the file from the upload location you can attempt a manual upload again.emctl upload agent Check the logfile again for errors. you can repeat for subsequent bad xml files, but if all else fails we can always clear the agent completely. Keep in mind you will lose data from the pending xmls therefore the data will not make it to the Grid Control Repository.
Clear the agent up

1. Stop the agent on the target node
 

emctl stop agent
2. Delete any pending upload files from the agent home
 

rm -r $ORACLE_HOME/sysman/emd/state/*
rm -r $ORACLE_HOME/sysman/emd/collection/*
rm -r $ORACLE_HOME/sysman/emd/upload/*
rm $ORACLE_HOME/sysman/emd/lastupld.xml
rm $ORACLE_HOME/sysman/emd/agntstmp.txt
rm $ORACLE_HOME/sysman/emd/blackouts.xml

3. agent clearstate
 

emctl clearstate agent

4. Start the agent again
 

emctl start agent

5. Force an upload to the Oracle Management Server/Service (OMS)
 

emctl upload agent

There could be another reason when you delete an agent and try to reconfigure it in OMS. Issue arise when previous delete is not completed successfully.

You can have following error messages.

EMD upload error: Failed to upload file A0000001.xml: Fatal Error.
Response received: 500|ORA-20618: The specified agent is in the process of being deleted from the repository, wait for deletion to complete before restarting the agent.(agent name = dbp1405.xx.xx.xxxxx.xxx:1830)(agent guid = 35F8188A75ABF6AFAAA6871B64D1C0B1)

ORA-06512: at "SYSMAN.TARGETS_INSERT_TRIGGER", line 30
ORA-04088: error during execution of trigger 'SYSMAN.TARGETS_INSERT_TRIGGER'

This is due to duplicate entry in OMS database.

I tried to delete that entry manually in OMS database by connecting as SYSMAN user

SQL> exec mgmt_diag.PurgeOrphanTarget(HEXTORAW('35F8188A75ABF6AFAAA6871B64D1C0B1'));
BEGIN mgmt_diag.PurgeOrphanTarget(HEXTORAW('35F8188A75ABF6AFAAA6871B64D1C0B1')); END;

*
ERROR at line 1:
ORA-20000: Target is in pending delete state
ORA-06512: at "SYSMAN.MGMT_DIAG", line 1437
ORA-06512: at line 1

 This means that it can not be deleted manually due to previous attempt still running. In order to find when it started then I ran this query

select delete_request_time, delete_complete_time, last_updated_time from mgmt_targets_delete where target_name='dbp1405.xx.xx.xxxx.xxx:1830' AND target_type='oracle_emd';

DELETE_REQUEST_TIME           DELETE_COMPLETE_TIME          LAST_UPDATED_TIME
----------------------------- ----------------------------- -----------------------------
01-jun-2012 11:48:36                                        01-jun-2012 11:48:36

  

This means that previous deletion hung for a long time.


There are couple  methods of progressing it further from this stage is to use repvfy utility to remove duplicate host from repository. If repvfy won't work then you can remove it manually by using the following steps.

Stop the agent

emctl stop agent

Then login to repository database using SYSMAN user the find the details about GUID for that object from MGMT_TARGETS_DELETE  table or use this statement.

SQL> select TARGET_NAME,TARGET_TYPE,EMD_URL from MGMT_TARGETS_DELETE WHERE TARGET_GUID='35F8188A75ABF6AFAAA6871B64D1C0B'

Check for all below tables:

Main tables are:

MGMT_TARGETS_DELETE
MGMT_EMD_PING
MGMT_EMD_PING_CHECK
MGMT_AVAILABILITY
MGMT_AVAILABILITY_MARKER
MGMT_CURRENT_AVAILABILITY

Also some reference tables,request you to check the below tables also:

MGMT_AGENT_SEC_INFO
MGMT_BLACKOUT_PROXY_TARGETS
MGMT_CURRENT_METRICS
MGMT_LAST_VIOLATION
MGMT_METRICS_1DAY
MGMT_POLICY_ASSOC_EVAL_DETAILS
MGMT_POLICY_ASSOC_EVAL_SUMM
MGMT_PURGE_POLICY_TARGET_STATE
MGMT_RT_BOOTSTRAP_TIMES
MGMT_STRING_METRIC_HISTORY
MGMT_TARGET_PROPERTIES
MGMT_VIOLATIONS


If in any table you find the entry for agent dbp1405.nl.eu.abnamro.com so please delete the target

example:

SQL> delete from MGMT_TARGETS_DELETE where TARGET_GUID='35F8188A75ABF6AFAAA6871B64D1C0B'

This should delete the duplicate or old entry from OMS and allow you to continue securing and uploading agent.