Mini Tutorial: Using Python to Access the OCI REST API (without using OCI_REST utils)

Open your Python app:

 

The following example simply gets the number of storage arrays on the OCI server ociserver.corp.com:
 
import requests
from base64 import b64encode
import json
 
baseurl = "https://ociserver.corp.com/rest/v1"
 
userAndPass = b64encode(b"yourRestApiUser:Password").decode("ascii")
headers = {'Authorization':'Basic %s' % userAndPass}
 
response = requests.request("GET",baseurl + "/assets/storages/count", headers = headers, verify = False)
 
response
response.text
response.json()
NumberOfStorages = response.json()['value']
print("Number of Storages = " + str(NumberOfStorages))
 
A simple function that does the request and returns the json() output.
 
def easyOciGet(path):
  response = requests.request("GET",baseurl + path, headers = headers, verify = False)
  return response.json()
 
easyOciGet('/assets/storages/count')
 
To get rid of the ‘InsecureRequestWarning’.
  
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
 
easyOciGet('/assets/storages/count')
 
Listing storages.
  
storages = easyOciGet('/assets/storages')
 
storages
print(storages)
 
for storage in storages: print(storage)
for storage in storages: print(storage['name'])
for storage in storages: print(storage['name'].ljust(25, ' ') + storage['model'].ljust(25, ' ') + storage['id'])
 
Listing a storage.
 
easyOciGet('/assets/storages/1234')
 
All pools on a single storage.
 
easyOciGet('/assets/storages/1234/storagePools')
 
Storage nodes on a storage.
 
easyOciGet('/assets/storages/1234?expand=storageNodes')
 
Performance on a storage.
 
easyOciGet('/assets/storages/1234?expand=performance')
 
Everything you can expand a storage by.
 
easyOciGet('/assets/storages/1234?expand=_expands')
 
Other examples:

• easyOciGet('/assets/storages/1234?expand=_expands,storagePools')
• easyOciGet('/assets/storages/1234?expand=storagePools.volumes')
• easyOciGet('/assets/storages/1234?expand=storagePools.volumes.performance')
• easyOciGet('/assets/storages/1234?expand=performance.history')

 
Other things you can do with the NetApp OCI REST API request:
 
Can filter on times using fromTime and toTime (these are time stamps in milliseconds in the UNIX epoch - check out https://www.unixtimestamp.com/ * 1000):
 
?expand=performance.history&fromTime=XX&toTime=YY
 
Expand, query, sort, fields, limit, offset:
  
/rest/v1/assets/volumes/{id}?expand=masks.host,masks.storagePort
/rest/v1/query?objectType=Volume&sort=storage.name&fields=masks.host,masks.storagePort&limit=5&offset=0

[Python] Translating Curl into Python

I needed to Python-ize some code I had written for a Delphix Reporting integration with NetApp’s OCI. This KB for Delphix Reporting uses curl to login to the API and get data:

https://support.delphix.com/Delphix_Reporting/KBA1734_Delphix_Reporting_(Formerly_Mission_Control)_Web_Service_API_Guide

curl --data "password=YOURPASS&user=YOURUSER" http://YOURSERVER/api/login

 

Note: https://curl.trillwords.com is a nice online curl to Python converter (do not enter real data; just use it to get the syntax right.)

 

Translated into Python, the above should be:

 

import requests

 

data = {

  'password': 'YOURPASS',

  'user': 'YOURUSER'

}

 

response = requests.post('https://YOURSERVER/api/login', data=data)

 

Unfortunately, I don’t have success when I try to connect to Delphix Reporting.

 

ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)

 

I had to put verify=False at the end, like so.

 

import requests

 

data = {

  'password': 'YOURPASS',

  'user': 'YOURUSER

}

 

response = requests.post('https://YOURSERVER/api/login', data=data, verify=False)

 

Great, we have our response but how do we access our required output (loginToken & userId)?

 

import json

json_data = response.json()

loginToken = json_data["loginToken"]

userId = json_data["userId"]

 

Great, so now we know how to get our loginToken and userId. How do we get Storage Summary data?

 

From the KB:

 

curl -H "X-Login-Token: LOGINTOKEN" -H "X-User-Id: USERNAME" "http://YOURSERVER/api/get_report?report=result_storage_summary"

 

This translates to:

 

headers = {

  'X-login-token': loginToken,

  'X-user-id': userId

}

 

params = (

  ('report','result_storage_summary'),

)

 

response = requests.get('https://YOURSERVER/api/get_report', headers=headers, params=params, verify=False)

 

Note: Remember the False in verify=False is case sensitive – it must have a capital F.

 

And this worked. To get the output into JSON format, run:

 

json_data = response.json()

 

And to see the JSON format output, run:

 

json_data

How to Find a Specific Column Name (e.g Custom Annotation) in the NetApp OCI DWH

The following is quite a nice query if you're trying to find everywhere a particular column name appears in the NetApp OnCommand Insight Data Warehouse MySQL database. Perhaps you are looking for a custom annotation, and want to find the table(s) that annotation presents itself in. The SQL query searches through all column names, in all tables, in all schemas, in the OCI DWH. In the example below we use country as the custom annotation we are searching for (the search is not case sensitive.)

SELECT TABLE_SCHEMA,TABLE_NAME,COLUMN_NAME,COLUMN_COMMENT FROM

(SELECT * FROM information_schema.columns WHERE table_schema = 'dwh_capacity'

UNION SELECT * FROM information_schema.columns WHERE table_schema = 'dwh_capacity_efficiency'

UNION SELECT * FROM information_schema.columns WHERE table_schema = 'dwh_cloud_cost'

UNION SELECT * FROM information_schema.columns WHERE table_schema = 'dwh_custom'

UNION SELECT * FROM information_schema.columns WHERE table_schema = 'dwh_fs_util'

UNION SELECT * FROM information_schema.columns WHERE table_schema = 'dwh_inventory'

UNION SELECT * FROM information_schema.columns WHERE table_schema = 'dwh_performance'

UNION SELECT * FROM information_schema.columns WHERE table_schema = 'dwh_ports'

UNION SELECT * FROM information_schema.columns WHERE table_schema = 'dwh_reports') as t0

WHERE COLUMN_NAME = 'Country'; -- for example...

Image: Search all Columns in all Tables in all Schemas (MySQL)

NetApp DWH SQL: Find Latest Updated/Created Tables in dwh_custom and test

Along similar lines, something else that is useful if you've inherited a DWH SQL instance with many tables in dwh_custom or test that you're not sure what they're for or if they are even used any more.

select TABLE_SCHEMA,TABLE_NAME,CREATE_TIME,UPDATE_TIME,TABLE_COMMENT from information_schema.tables where TABLE_SCHEMA = 'dwh_custom' order by UPDATE_TIME desc;

select TABLE_SCHEMA,TABLE_NAME,CREATE_TIME,UPDATE_TIME,TABLE_COMMENT from information_schema.tables where TABLE_SCHEMA = 'dwh_custom' order by CREATE_TIME desc;

select TABLE_SCHEMA,TABLE_NAME,CREATE_TIME,UPDATE_TIME,TABLE_COMMENT from information_schema.tables where TABLE_SCHEMA = 'test' order by UPDATE_TIME desc;

select TABLE_SCHEMA,TABLE_NAME,CREATE_TIME,UPDATE_TIME,TABLE_COMMENT from information_schema.tables where TABLE_SCHEMA = 'test' order by CREATE_TIME desc;

Python 3: OCI REST API - Set Annotation On Storage Array

In the NetApp OCI API Samples for Python (available from the OCI Operational Server WebUI > REST API documentation > API samples) there is a set_annotation.py which can be used to set the annotation on a specific volume. With a slight modification of the Python, we get a set_annotation_storage.py which can be used to set the annotation on a specific storage. An example of using it>

python set_annotation_storage.py --url https://YOUR_OCI_SERVER --user YOUR_REST_USER --password YOUR_PASSWORD --storage STORAGE_NAME -annotation SOME_ANNOTATION --value ANNOTATION_VALUE

Image: Insight REST API documentation > API samples

The Script

Save as say ‘set_annotation_storage.py’.

#!/usr/bin/env python 

"""

Set annotation for a specific storage

"""

from __future__ import print_function

import json

from oci_rest import OciRest, configure_command_line_parser

def find_storage(oci, storage_name):

  for storage in oci.get('assets/storages'):

    if storage['name'] == storage_name:

      return storage['self']

  return None

def annotate_storage(oci, storage_url, annotation_name, annotation_value):

  return oci.put('{}/annotations'.format(storage_url), data=json.dumps(

    [

      {

        "rawValue": annotation_value,

        "definition": {"name": annotation_name}

      },

    ]

  ))

if __name__ == "__main__":

  # Get the default command-line arguments (url, user, password)

  parser = configure_command_line_parser(usage='Set storage annotation')

  

  # Add additional arguments to configure the search criteria

  parser.add_argument('--storage', required=True, help="Storage name")

  parser.add_argument('--annotation', required=True, help="Annotation name (do NOT use label!)")

  parser.add_argument('--value', required=True, help="Annotation value to set")

  

  options = parser.parse_args()

  url, user, password = options.url, options.user, options.password

  

  with OciRest(url, user, password) as oci:

    storage_url = find_storage(oci, storage_name=options.storage)

    

    if storage_url is not None:

      new_annotation = annotate_storage(oci,

          storage_url,

          annotation_name=options.annotation,

          annotation_value=options.value)

      

      print('Created new annotation:\n', json.dumps(new_annotation))

      

    else:

      print("Could not find storage {}".format(options.storage))

APPENDIX: List of the 14 out-of-the-box ‘NetApp OCI API Samples for Python’

• basic.py - print list of all storagePools with storage name and total allocatedCapacity
• annotation_type.py - manipulate annotation types
• business_entity.py - manipulate business entities
• collection_report.py - Print status report for each datasource
• create_user.py - Create OCI user
• filter_storagepool_by_thresholds.py - print list of storage pools filtered by storage tier, and thresholds
• filter_vms_by_storage.py - print virtual machines with dataStores on specific storage
• license.py - Programmatically manipulate OCI license
• import_patch.py - Import datasource patch
• set_annotation.py - Set annotation for a specific volume
• set_annotation_based_on_path.py - Set datacenter annotation for hosts based on their connected storage annotation
• set_annotation_bulk.py - Set annotation for a group of volumes identified with prefix
• set_ldap_config.py - set values for LDAP configuration
• integrations.py - create, delete agents and integrations. Ingest integration data.

Python 3: Basic Website Monitor

A bit like the multiple ping test monitor I wrote in PowerShell back in 2015. Here’s my Python 3: Basic Website Monitor.

Very simple to use, just save the script below as say websiteChecker.py and then run the script like -

python websiteChecker.py www.website1.com www.website2.com

- where the arguments are the website you want to monitor (no limits on the number of website you want to monitor.)

In the example below I’ve run -

python websiteChecker.py www.furruri.com www.porsche.com www.jagwar.com www.lambo.com

(with some intentional spelling mistakes)

Image: Example of the Python 3 Basic Website Monitor in action

If you missing some of the Python libraries, check out the previous post here

The Script

'''

websiteChecker.py

This program expects websites as arguments, and will add https:// to the name.

Example> python websiteChecker.py www.furrari.com www.porsche.com

'''

import sys

import time

import colorama

import urllib.request

from termcolor import *

colorama.init()

def webCheck(url):

  url = "https://" + url

  try:

    status_code = urllib.request.urlopen(url).getcode()

    return("OK(" + str(status_code) + ")")

  except urllib.error.URLError:

    return("FAILED")

# Create status array...

status = [''] * len(sys.argv)

x = 1

for webSite in sys.argv[1::]:

  status[x] = "UNTESTED"

  x += 1

chkNext = 1 # Website to check next

loop = "Y"

while loop:

  print(chr(27) + "[2J") # CLS

  x = 1

  for webSite in sys.argv[1::]:

    if status[x] == "UNTESTED":

      cprint(status[x].ljust(10) + " - " + sys.argv[x].ljust(25),'white','on_blue')

    if status[x].startswith("OK"):

      cprint(status[x].ljust(10) + " - " + sys.argv[x].ljust(25),'white','on_green')

    if status[x] == "FAILED":

      cprint(status[x].ljust(10) + " - " + sys.argv[x].ljust(25),'white','on_red')

    x += 1

  print("\nWebsite to check next:\n" + str(sys.argv[chkNext]))

  time.sleep(1)

  status[chkNext] = webCheck(sys.argv[chkNext])

  chkNext += 1

  if chkNext >= len(sys.argv):

    chkNext = 1

Python 3: Basic Test for Web Site Availability

  In Python 3, doing a basic test for web site availability is very straightforward:

• >>> import urllib.request
• >>> print(urllib.request.urlopen("https://www.cosonok.com").getcode())
• 200
• >>>

It needs a little modification so you don't get an ugly error if the website is not available:

• >>> try:
• ...     a = urllib.request.urlopen("https://www.cosonok.co").getcode()
• ...     print("OK (" + str(a) + ")")
• ... except urllib.error.URLError: print("FAILED")
• ...
• FAILED
• >>>

What if we want colors?

The below shows you how to install termcolor in the command prompt. There are two methods: 1) for if 'pip download' works for you, and 2) if you have to get the file manually from one machine to install on another machine. Because I am running Python on a Windows machine, we also need to get colorama. Windows cmd doesn't support ANSI coding like linux/unix which termcolor uses (perhaps termcolor is surplus to requirements with Windows cmd...)

Method 1:

• > python -m pip install termcolor
• > python -m pip install colorama

Method 2:

• python -m pip download termcolor
• python -m pip install termcolor-1.1.0.tar.gz
• pythom -m pip download colorama
• python -m pip install colorama-0.4.3-py2.py3-none-any.whl

Note: See https://pypi.org/project/termcolor/ for more on termcolor.

A modified Python script using colors - save this as say webCheck.py:

• import sys
• from termcolor import *
• import colorama
• colorama.init()
• import urllib.request
• 
  
• url = "https://www.cosonok.com"
• try:
•     status_code = urllib.request.urlopen(url).getcode()
•     cprint(("OK(" + str(status_code) + ")"),'white','on_green')
• except urllib.error.URLError:
• cprint("FAILED",'white','on_red')

Image: Example running webCheck.py

A very sinple example. To be enhanced in a future post...

NetApp OCI REST API Python Samples Requirements - On a Corporate Desktop?

From this post ‘NetApp OCI REST API and Python’ we see that it’s easy to use Python with NetApp OCI’s REST API. And this worked fine on my corporate desktop.

When I was looking in the ‘OCI REST API samples’ for Python (available for download from the OCI WebUI) - which utilizes a library to simplify access to the OCI REST API - oci_rest.py - the pre-requisites instructions include to run (from a command prompt since my desktop is Windows):

python -m pip install --requirement requirements.txt

Where requirements.txt is:

requests>=2.9.0

Unfortunately, running the above generates an error on my corporate desktop. Understandably, there is some corporate restriction in place that prevents me downloading the wheel files (I count myself lucky to have access to Python in the first place!)

The error I get is:

• Could not find a version that satisfies the requirement...
• No matching distribution found for requests>=2.9.0 ...

But when I run the same command on my home PC, it works and downloads 5 files as below:

• Downloading requests-2.24.0-py2.py3-none-any.whl (61 kB)
• Downloading certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
• Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
• Downloading urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
• Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)

Since these are not massive files, could I download them and rename them as txt files, then email in?

It is possible to download all the above files individually over the internet, but much easier for me to just use ‘pip download’ from my home PC, as below (from CMD):

python -m pip download --requirement requirements.txt

And then we’ll convert the extensions to .txt, ZIP the files, and see if we can email into the Enterprise.

Miracle of miracles, the ZIP arrives. I unzip and convert the extensions back to .whl files!

Image: NetApp OCI Python Requirements

Now to install them (again from CMD)!

Using trial and error, below is the correct order to install them:

Note: If you try to install in the order above, you’ll find some missing requirements, and see the same error as above. 

• python -m pip install idna-2.10-py2.py3-none-any.whl
• python -m pip install certifi-2020.6.20-py2.py3-none-any.whl
• python -m pip install urllib3-1.25.10-py2.py3-none-any.whl
• python -m pip install chardet-3.0.4-py2.py3-none-any.whl
• python -m pip install requests-2.24.0-py2.py3-none-any.whl

YES! JOB DONE! Now to utilize the OCI REST API Python examples!

To use the NetApp OCI Python REST API examples, the first thing you'll need to do from you command line is>

python oci_rest.py install

Then in your Python shell/code, you'll want:

from oci_rest import *

Note: The above imports everything. Best practice is to be more selective.