vCOPs: Error 500 The call failed on the server; see server log for details (StatusCode: 500)

Error 500 is rather annoying and quite generic “catch-all” error handler  for Web servers. VROPS is no different in this context.

In this blog , I aim to resolve one of the conditions with Analytics component of VROPS . Check if the symptoms and logs are relevant before you go ahead with the steps.

Symptoms

1- Login through ssh to the admin page of vrops

http://ipaddressofUIVM/admin

2- Check the status of all the services

Analytics

In this case you observe that VCOPS Manager Service & Analytics Service stopped

3-This issue may also manifest as below , but you have not changed the ip address of either of the VMs(UI or Analytics)

The UI VM running at <IP address> but cannot to the Analytics VM at <IP address >, make sure it is running and reachable from <IP address> If the IP address of either VM has changes, then login to Administration interface that will guide you through the steps to restore connectivity between the two VMs.

 

4- Check logs $ALIVE_BASE/user/log/analytics.log

2015-11-12 16:38:41,225 ERROR [Thread-1] com.integrien.alive.dbaccess.AnalyticsFastLoaderCache.loadActiveAlarmsNative – Error while loading active alarms org.postgresql.util.PSQLException: PANIC: checksum mismatch: disk has 0x41ea3421, should be 0xda4b0281 filename pg_tblspc/16385/PG_9.0_201106101/16386/16435, BlockNum 171234, block specifier 16385/16386/16435/0/171234

2015-11-12 16:38:42,245 INFORMATION [Thread-1] com.integrien.analytics.AnalyticsMain.stop – Analytics is stopping
2015-11-12 16:38:42,265 INFORMATION [Thread-1] com.integrien.analytics.AnalyticsMain.stop – AnalyticsService has been stopped  

If these symptoms match, proceed to the next steps,

Root cause : For some reason the postgres db used by the analytics engineer encountered an error and needs you to correct it.

Resolution

1- Login to Analytics VM

2- Connect to the DB

su postgres

3- Execute ” pg_ctl” stop -m smart -D /data/pgsql/data

** Force it using immediate switch if required/previous command errors out

4- Execute the following command and replace bolded text with the actual tablespace/database/relation/fork/blockNum recorded in analytics log

postgres –single -D $PGDATA -c fix_block_checksum= “16385/16386/16435/0/171234″

This should return with an message as ” ..* fixed”

5- pg_ctl start -D /data/pgsql/data

6- Go ahead and restart the vcops services

Happy Monitoring …

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s