vCOPs: Error 500 The call failed on the server; see server log for details (StatusCode: 500)

Error 500 is rather annoying and quite generic “catch-all” error handler  for Web servers. VROPS is no different in this context.

In this blog , I aim to resolve one of the conditions with Analytics component of VROPS . Check if the symptoms and logs are relevant before you go ahead with the steps.

Symptoms

1- Login through ssh to the admin page of vrops

http://ipaddressofUIVM/admin

2- Check the status of all the services

Analytics

In this case you observe that VCOPS Manager Service & Analytics Service stopped

3-This issue may also manifest as below , but you have not changed the ip address of either of the VMs(UI or Analytics)

The UI VM running at <IP address> but cannot to the Analytics VM at <IP address >, make sure it is running and reachable from <IP address> If the IP address of either VM has changes, then login to Administration interface that will guide you through the steps to restore connectivity between the two VMs.

 

4- Check logs $ALIVE_BASE/user/log/analytics.log

2015-11-12 16:38:41,225 ERROR [Thread-1] com.integrien.alive.dbaccess.AnalyticsFastLoaderCache.loadActiveAlarmsNative – Error while loading active alarms org.postgresql.util.PSQLException: PANIC: checksum mismatch: disk has 0x41ea3421, should be 0xda4b0281 filename pg_tblspc/16385/PG_9.0_201106101/16386/16435, BlockNum 171234, block specifier 16385/16386/16435/0/171234

2015-11-12 16:38:42,245 INFORMATION [Thread-1] com.integrien.analytics.AnalyticsMain.stop – Analytics is stopping
2015-11-12 16:38:42,265 INFORMATION [Thread-1] com.integrien.analytics.AnalyticsMain.stop – AnalyticsService has been stopped  

If these symptoms match, proceed to the next steps,

Root cause : For some reason the postgres db used by the analytics engineer encountered an error and needs you to correct it.

Resolution

1- Login to Analytics VM

2- Connect to the DB

su postgres

3- Execute ” pg_ctl” stop -m smart -D /data/pgsql/data

** Force it using immediate switch if required/previous command errors out

4- Execute the following command and replace bolded text with the actual tablespace/database/relation/fork/blockNum recorded in analytics log

postgres –single -D $PGDATA -c fix_block_checksum= “16385/16386/16435/0/171234″

This should return with an message as ” ..* fixed”

5- pg_ctl start -D /data/pgsql/data

6- Go ahead and restart the vcops services

Happy Monitoring …

 

A Network Virtualization fairy tale . . .

I recently had the opportunity to co-present at vForum Singapore 2015 along with Al Da Silva from Juniper on the topic – “Smart Software is Indispensable, Smart Networking Hardware is Fundamental”

The flow of the presentation starts with the Value of SDDC and why it is the way forward.

We then provided a basic overview of what NSX is all about, while at it we also debunked some questions and myths around Network Virtualization such as , “Are we replacing Network hardware through NSX ?” or “Are the network engineers’ career doomed !!!”

Not really, to understand better draw a parallel of what was done in the Server Virtualization space. We consolidated x86 servers into more powerful hardware and enabled phenomenal features like vMotion, HA , DRS that transformed the way servers were managed.

By doing so we met the following objectives – Capex/Opex reduction, better resiliency, better operations management and automation capabilities and most importantly we delivered IT as a Service in the most optimal, efficient and rapid pace.

Very similarly from a networking standpoint, we want to be able to implement Network & Security Services such as load balancing, firewall, switching & routing from the software layer to accomplish the same objectives. We obviously are not getting rid of the physical networking gear, we actually need the IP backplane to be able to scale and optimize to meet the requirements of a Software Defined Networking.

We got into more details as we spoke about the convergence of Overlay and Underlay networks. We discussed tunneling as an example to demonstrate how different VxLAN replication modes require the physical network to be configured in a certain way to be able to build a robust L2 over L3. Juniper is able to provide “Single Switch management ” capabilities via Virtual Chassis Fabric. This is being positioned as the ideal fabric layout for the SDDC.

We then delved into how SDDC created newer challenges that were non-existent in conventional datacenter, for instance in a multi-tenant cloud how we cannot allow IP workloads run adjacently i.e. on the same ESX host without sufficient protection. This obviously requires security at the Virtual NIC level. With Micro-segmentation we are able to achieve just that, traffic is intercepted and governed even before it hits the “Virtual Wire”.

My favorite part of the session was the Management integration,VROPS NSX Management Pack together with Juniper Network Director provide amazing management and monitoring capabilities. For instance a drop down menu has a list of overlay identifiers(VNI), at the click of the VNI we can see all the VM’s running across various different hosts and their datapaths. Troubleshooting made easy isn’t it.

I’ve attached the presentation deck for your reading pleasure, it’s mostly self explanatory, but obviously not the same as being part of the live session.

If there is any ambiguity, drop me a note – cedric.rajendran@gmail.com

VMW_Juniper

..a true fairy tale in the technology arena

Cheers,

Cedric

Getting Started with VMware Virtual VSAN

“Why do we have to listen to our hearts?” the boy asked, “Because, wherever your heart is, that is where you’ll find your treasure.”

– “The Alchemist” by Paulo Coelho

In pursuit of my dreams, I debut as an author.

A blurb about the book,

Title :  “Getting started with VSAN”

Build an optimal, high performance and resilient software defined storage on VSAN for your vSphere infrastructure

Objectives

  • Effectively understand the fundamental concepts and features of Virtual SAN
  • Implement and administer your VMware VSAN efficiently
  • Ensure stability, data reliability and meet service level agreements for each virtual machine and application

Summary
Virtual SAN without any doubt is the latest fad in the Virtualization Arena. The concept has taken Datacenter Virtualization to the next level by moving the storage back from SAN to server hardware. The best part of VSAN solution is the fact that a near flash performance is achieved with only a marginal percentage of flash storage. And all this with hardly any compromise to the exquisite feature set of vSphere and interoperability with other product suite from VMware. This book helps in closely examining the capabilities as well the limits of VSAN in a sequential flow while explaining the fundamental aspects.

Description
This book covers the fundamental aspects of Software Defined storage, its evolution and its role in a Software-Defined Data Center. You will have a complete understanding of VMware Virtual SAN architecture, its components and how it is put together. Guidelines pertaining to sizing and profiling of Virtual Machines are also covered in detail.

The book explores your options in choosing the type of Virtual SAN and deploys a simple setup that would demonstrate to you the workflow of a VSAN deployment. You will review policy based management of storage and see how Storage Policy-Based Management modernizes storage provisioning and paves the way for being truly software defined.

After covering the anatomy of I/O-Input/Output to a VSAN datastore, some insight into design considerations and best practices of VSAN configuration, the book ends with a discussion on troubleshooting and how various utilities can be leveraged to diagnose common problems that surface on a Virtual SAN environment.

What you will learn

  • Understand what Software Defined Storage is and how it is implemented through VMware Virtual SAN
  • Install, Configure and Administer VSAN
  • Architect VSAN to cater to the specific business requirements
  • Define Storage Tiering at the software layer by implementing Storage Policy-Based Management
  • Understand how every I/O flows through the VSAN infrastructure
  • Monitor your VSAN Infrastructure to discover issues at an early stage
  • Efficiently use utilities to troubleshoot and fix issues

Who this book is written for

This book is intended for Server Administrators and Storage Administrators who would like to successfully build and scale a VSAN backed vSphere infrastructure. Basic understanding of vSphere concepts and storage fundamentals will be helpful.

Style and approach

A simple step-by-step guide to walk you in-depth through software defined storage and virtual VSAN. Each chapter is written keeping in mind that the audience are not particularly storage experts rather server administrators.

Where can you get it

The book is available @
https://www.packtpub.com/virtualization-and-cloud/getting-started-vmware-virtual-san

Do feel free to write to me with your feedback at cedric.rajendran@gmail.com

Cheers !!!