Wednesday, 30 December 2020

Goodbye 2020 and Hello 2021!

Wishing You a Healthy and Prosperous 2021!

Or ‘Live Long and Prosper’ as Mr Spock would say!

Or ‘Peace and Long Life’ as Mr Spock would say if you said to him ‘Live Long and Prosper’!


It has been a very strange year 2020...

.
.
.
.
.
.
.

And that was going to be my end of year 2020 post...

One thing I'll share...

My greatest discovery of 2020!

Check out the works of Alan Watts - some places to start:


Blog Stats in 2020

  •  Cosonok’s IT blog: 248k views (Total Pageviews = 3,423,579)
  • My other IT blog*: 26.7k views (Total Pageviews = 148,211)
  • My car blog: 3.22k views
  • My philosophy blog: 1.78k views

*I do not broadcast my other IT blog (not much anyway). It’s turned into more of a note-y type blog, stuff which in the past I might have added to the ‘Brief Notes’ page (which got too big.) My other IT blog has had a good year, over 120 new posts (compared to just 55 for my main blog.)

How Much Money Have I Made from Blogging All Time?

Something I have never shared is how much money I have made from allowing adverts on my blog (which I think you should allow, as that’s how blogger provides their excellent free service). So, if you are curious how much money you could make from a 3 million + pageviews blog, look below.  

  • Sep 26, 2012: £62.31
  • Dec 28, 2012: £73.93
  • Apr 24, 2013: £78.19
  • Oct 22, 2013: £71.61
  • Jan 22, 2014: £63.14
  • May 28, 2014: £84.39
  • Aug 28, 2014: £60.85
  • Nov 26, 2014: £73.81
  • Feb 23, 2015: £64.06
  • May 27, 2015: £65.55
  • Aug 25, 2015: £63.51
  • Dec 23, 2015: £70.44
  • May 25, 2016: £66.67
  • Nov 21, 2016: £65.98
  • Apr 26, 2017: £64.92
  • Oct 25, 2017: £65.17
  • May 22, 2018: £71.2
  • Jun 25, 2019: £63.36
  • Oct 23, 2020: £60.22
  • TOTAL = £1289.31

2014 and 2015 were very good years. The last 3 years have each seen just one payment.

.
.
.

Happy New Year!

Tuesday, 29 December 2020

NetApp Insight 2020: Cloud Insights Content

 

From the previous post Things To Do If You Have Free Time Over Christmas! I shared the links to the NetApp Insight 2020 content (Content Library or Panopto for NetApp Employees and Partners only).

I have spent a little time over the festive break going over the NetApp Cloud Insights content from NetApp Insight 2020. Below are relevant sessions, further resources, and some bonus items.

 

Cloud Insights NetApp Insight 2020 Content

Note: The timings are in the brackets.

  • MEGA-1676-2 More cloud. Less cost [18:06]
  • BRK-1224-2 ONTAP Management with the Power and Scale of the Cloud [31:51]
  • BRK-1289-2 Monitor Your Hybrid Cloud [43:32]
  • DEM-1315-1 End-to-End Monitoring for NetApp and Multicloud [9:29]
  • DEM-1518-2 Hybrid Cloud Management in NetApp Cloud Insights [11:37] (this demo appears in BRK-1289-2)
  • DEM-1350-2 Awesome Visualizations for NetApp Performance [9:56]
  • SPD-1327-3 Monitor Everything! DIY Agents in Cloud Insights [16:45] (collecting metrics with Telegraf)
  • SPD-1319-3 Integrate Data from ServiceNow into Infrastructure Management [18:50]

 

Cloud Insights with Other Products / Notable Mentions

  • Key-1632 Let's Talk Cloud: How is cloud changing the world? [56:49]
  • MEGA-1679-2 Contain your cabin fever with Kubernetes [19:54]
  • BRK-1229-1 Monitor It All: Spot Cloud Analyzer and Cloud Insights [36:03]
  • BRK-1401-1 Practical Approaches to Implementing Automation in a Hybrid Cloud [1:00:55]
  • BRK-1410-2 How To Build a Cloud-Like as-a-Service IT Model on Premises [40:03]
  • BRK-1378-2 Move to the Cloud and Improve Privacy, Security, and Compliance [42:25]
  • BRK-1328-2 Detect and Stop Ransomware Before It's Too Late [29:37]
  • BRK-1320-3 Find and Fix Your Kubernetes Infrastructure Issues, Fast! [39:55]
  • DEM-1534-1 NetApp AFF A250 Virtual Tour and Demo [7:26]
  • DEM-1517-2 Ransomware Detection in NetApp Cloud Insights [9:03]
  • DEM-1368-2 3D Cloud: Graphic Applications in the Cloud [41:09]
  • SPD-1466-1 Accelerate Your Digital Transformation Journey [18:41]
  • SPD-1427-1 NetApp Private Cloud: Solve for Today, Prepare for Tomorrow [20:42]
  • SPD-1165-1 Build a Better Private Cloud: Performance, Flexibility, TCO [15:58]
  • SPD-1387-2 Virtual Desktop Service: New Functionality Overview [15:30]
  • SPD-1299-2 The Dirty Secrets About Virtual Desktops [16:18]
  • SPD-1362-3 Seven Deadly Sins of EUC/VDI in the Cloud [22:23]

More automation related sessions: 1103 / 1295 / 1316 / 1385 / 1411 

Resources from DEM-1350-2: Awesome Visualizations for NetApp Performance

 

Resources from SPD-1327-3: Monitor Everything! DIY Agents in Cloud Insights

Image: Mapping OCI/CI Web UI to REST API

 

  • OCI: Replace ui/#assets with rest/v1/assets
  • CI: Replace web/# with rest/v1


Useful Additional Information


BONUS: Things You Can Do with Cloud Insights (Dashboards from a Cloud Insights Demo)

Monitoring Value
Monitoring - Assure your services

  • Web App KPIs - Full Stack
  • How does my HCI Environment perform?
  • NetApp HCI Lab Admin - How much headroom do I have?
  • Storage Admin - Which nodes are in high demand?
  • Storage Admin - Which pools are in high demand?
Cost Optimization Value
Cost Optimization - Reduction of Waste

  • Where do I have wasted resources?
  • Virtual Reclamation and Remediation - AWS
  • EBS Volume Performance Optimization
  • On-Prem Storage Optimization
  • Am I Getting The Most From My ONTAP Efficiencies?
Troubleshooting Value
Troubleshooting - Quickly Identifying Where Issues Are

  • What Policy Violations Exist in my Environment?
  • Where Are My Shared Application Resources impacted?
  • When is my NetApp Storage Node Headroom Not Meeting Optimal Utilization?

Dashboards (Full List):  Agent Node Monitoring - Acquisition Units / Application Servers Dashboard / AWS Admin - Where can I reclaim waste?  / AWS Storage Consumers  / Cloud Volumes ONTAP - Node Operational/Optimal Points / Cloud Volumes ONTAP - PrePost Efficiencies / Data Services by Organization / Data Services Dashboard / Infrastructure Usage / Monitoring - Web App KPIs - Full Stack / Monitoring - AWS Admin - Which EC2 are in high demand? / Monitoring - Cloud Volumes Ontap - How is Front End (ONTAP) and Back End (EC2) Performance? / Monitoring - data capacity plus snapshot equals total capacity / Monitoring - How does my HCI Environment perform? / Monitoring - Morning Coffee - Cloud Ops / Monitoring - NetApp HCI Lab Admin - How much headroom do I have? / Monitoring - Overall VM Performance / Monitoring - Storage Admin - Which nodes are in high demand? / Monitoring - Storage Admin - Which pools are in high demand? / Optimize - Am I Getting The Most From My ONTAP Efficiencies? / Optimize - EBS Volume Performance / Optimize - On-Prem Storage Optimization / Optimize - Virtual Reclamation and Remediation - AWS / Optimize - Where do I have wasted resources? (Capacity Focus) / Optimize - Where is CVO Storage Efficiency Helping Me? / Storage Admin - Which pools are in high demand? / Storage Operations Morning Coffee (All Vendors) / Tiering - Efficiencies Achieved per Tier / Tiering - Storage Pool Capacities and OverCommit by Storage and Tier / Tiering - Storage Tier Capacity and Performance / Troubleshooting - What Policy Violations Exist in my Environment? / Troubleshooting - When is my NetApp Storage Node Headroom Not Meeting Optimal Utilization? / Troubleshooting - Where Are My Shared Application Resources Impacted? / VMware Admin - Where are opportunities to right size? / VMware Admin - Where can I potentially reclaim waste?


Useful link for NetApp Employees:
https://netapp.sharepoint.com/sites/CloudDataServices/CloudInsightsProduct/SitePages/Home.aspx
Links to:

  • Cloud Insights Field Enablement
  • Cloud Insights Marketing Documents

Wednesday, 2 December 2020

Things To Do If You Have Free Time Over Christmas!

2020 has been a bit of a quiet year on this blog. 2020 has been a very crazy year. I've been lucky to be very busy with work. Just that the work I've been busy with doesn't make good blog content for this blog, or is not easy/possible to blog about. Also my "Tech Roundup's" have fallen by the wayside (they took time to compile and I never got time to go over the links myself.) Because of the 2020 madness, I have nearly 2 weeks off over Christmas (usually I take holiday earlier in the year), and it's a chance to catch up on some of the things I didn't get to do in 2020, whilst at the same time chilling out and having a merry time!

1) NetApp Insight 2020 - Content Library:
https://insightdigital.netapp.com/content-library


Vision Keynotes, Mega Sessions, and more!
300+ sessions and breakouts available in the content library.

2) Catch up with the Tech ONTAP Podcasts:
https://soundcloud.com/techontap_podcast

3) Take some training courses in areas that interest you! From - for example:

https://www.pluralsight.com/
https://www.udemy.com/
https://acloudguru.com/
https://www.netapp.com/support-and-training/netapp-university/
https://labondemand.netapp.com/

If it interests you, try googling "Best Python Game Course" or "Best DevOps Course".

Also - if you have access to them - the NetApp TechMaster content is great too.

Sunday, 25 October 2020

Mini Tutorial: Using Python to Access the OCI REST API (without using OCI_REST utils)

Open your Python app:

 



The following example simply gets the number of storage arrays on the OCI server ociserver.corp.com:
 
import requests
from base64 import b64encode
import json
 
baseurl = "https://ociserver.corp.com/rest/v1"
 
userAndPass = b64encode(b"yourRestApiUser:Password").decode("ascii")
headers = {'Authorization':'Basic %s' % userAndPass}
 
response = requests.request("GET",baseurl + "/assets/storages/count", headers = headers, verify = False)
 
response
response.text
response.json()
NumberOfStorages = response.json()['value']
print("Number of Storages = " + str(NumberOfStorages))
 
A simple function that does the request and returns the json() output.
 
def easyOciGet(path):
  response = requests.request("GET",baseurl + path, headers = headers, verify = False)
  return response.json()
 
easyOciGet('/assets/storages/count')
 
To get rid of the ‘InsecureRequestWarning’.
  
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
 
easyOciGet('/assets/storages/count')
 
Listing storages.
  
storages = easyOciGet('/assets/storages')
 
storages
print(storages)
 
for storage in storages: print(storage)
for storage in storages: print(storage['name'])
for storage in storages: print(storage['name'].ljust(25, ' ') + storage['model'].ljust(25, ' ') + storage['id'])
 
Listing a storage.
 
easyOciGet('/assets/storages/1234')
 
All pools on a single storage.
 
easyOciGet('/assets/storages/1234/storagePools')
 
Storage nodes on a storage.
 
easyOciGet('/assets/storages/1234?expand=storageNodes')
 
Performance on a storage.
 
easyOciGet('/assets/storages/1234?expand=performance')
 
Everything you can expand a storage by.
 
easyOciGet('/assets/storages/1234?expand=_expands')
 
Other examples:

  • easyOciGet('/assets/storages/1234?expand=_expands,storagePools')
  • easyOciGet('/assets/storages/1234?expand=storagePools.volumes')
  • easyOciGet('/assets/storages/1234?expand=storagePools.volumes.performance')
  • easyOciGet('/assets/storages/1234?expand=performance.history')
 
Other things you can do with the NetApp OCI REST API request:
 
Can filter on times using fromTime and toTime (these are time stamps in milliseconds in the UNIX epoch - check out https://www.unixtimestamp.com/ * 1000):
 
?expand=performance.history&fromTime=XX&toTime=YY
 
Expand, query, sort, fields, limit, offset:
  
/rest/v1/assets/volumes/{id}?expand=masks.host,masks.storagePort
/rest/v1/query?objectType=Volume&sort=storage.name&fields=masks.host,masks.storagePort&limit=5&offset=0

Tuesday, 6 October 2020

[Python] Translating Curl into Python

I needed to Python-ize some code I had written for a Delphix Reporting integration with NetApp’s OCI. This KB for Delphix Reporting uses curl to login to the API and get data:

https://support.delphix.com/Delphix_Reporting/KBA1734_Delphix_Reporting_(Formerly_Mission_Control)_Web_Service_API_Guide

curl --data "password=YOURPASS&user=YOURUSER" http://YOURSERVER/api/login

 

Note: https://curl.trillwords.com is a nice online curl to Python converter (do not enter real data; just use it to get the syntax right.)

 

Translated into Python, the above should be:

 

import requests

 

data = {

  'password': 'YOURPASS',

  'user': 'YOURUSER'

}

 

response = requests.post('https://YOURSERVER/api/login', data=data)

 

Unfortunately, I don’t have success when I try to connect to Delphix Reporting.

 

ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)

 

I had to put verify=False at the end, like so.

 

import requests

 

data = {

  'password': 'YOURPASS',

  'user': 'YOURUSER

}

 

response = requests.post('https://YOURSERVER/api/login', data=data, verify=False)

 

Great, we have our response but how do we access our required output (loginToken & userId)?

 

import json

json_data = response.json()

loginToken = json_data["loginToken"]

userId = json_data["userId"]

 

Great, so now we know how to get our loginToken and userId. How do we get Storage Summary data?

 

From the KB:

 

curl -H "X-Login-Token: LOGINTOKEN" -H "X-User-Id: USERNAME" "http://YOURSERVER/api/get_report?report=result_storage_summary"

 

This translates to:

 

headers = {

  'X-login-token': loginToken,

  'X-user-id': userId

}

 

params = (

  ('report','result_storage_summary'),

)

 

response = requests.get('https://YOURSERVER/api/get_report', headers=headers, params=params, verify=False)

 

Note: Remember the False in verify=False is case sensitive – it must have a capital F.

 

And this worked. To get the output into JSON format, run:

 

json_data = response.json()

 

And to see the JSON format output, run:

 

json_data


Saturday, 12 September 2020

How to Find a Specific Column Name (e.g Custom Annotation) in the NetApp OCI DWH

The following is quite a nice query if you're trying to find everywhere a particular column name appears in the NetApp OnCommand Insight Data Warehouse MySQL database. Perhaps you are looking for a custom annotation, and want to find the table(s) that annotation presents itself in. The SQL query searches through all column names, in all tables, in all schemas, in the OCI DWH. In the example below we use country as the custom annotation we are searching for (the search is not case sensitive.)

SELECT TABLE_SCHEMA,TABLE_NAME,COLUMN_NAME,COLUMN_COMMENT FROM

(SELECT * FROM information_schema.columns WHERE table_schema = 'dwh_capacity'

UNION SELECT * FROM information_schema.columns WHERE table_schema = 'dwh_capacity_efficiency'

UNION SELECT * FROM information_schema.columns WHERE table_schema = 'dwh_cloud_cost'

UNION SELECT * FROM information_schema.columns WHERE table_schema = 'dwh_custom'

UNION SELECT * FROM information_schema.columns WHERE table_schema = 'dwh_fs_util'

UNION SELECT * FROM information_schema.columns WHERE table_schema = 'dwh_inventory'

UNION SELECT * FROM information_schema.columns WHERE table_schema = 'dwh_performance'

UNION SELECT * FROM information_schema.columns WHERE table_schema = 'dwh_ports'

UNION SELECT * FROM information_schema.columns WHERE table_schema = 'dwh_reports') as t0

WHERE COLUMN_NAME = 'Country'; -- for example...


Image: Search all Columns in all Tables in all Schemas (MySQL)


NetApp DWH SQL: Find Latest Updated/Created Tables in dwh_custom and test

Along similar lines, something else that is useful if you've inherited a DWH SQL instance with many tables in dwh_custom or test that you're not sure what they're for or if they are even used any more.

select TABLE_SCHEMA,TABLE_NAME,CREATE_TIME,UPDATE_TIME,TABLE_COMMENT from information_schema.tables where TABLE_SCHEMA = 'dwh_custom' order by UPDATE_TIME desc;

select TABLE_SCHEMA,TABLE_NAME,CREATE_TIME,UPDATE_TIME,TABLE_COMMENT from information_schema.tables where TABLE_SCHEMA = 'dwh_custom' order by CREATE_TIME desc;

select TABLE_SCHEMA,TABLE_NAME,CREATE_TIME,UPDATE_TIME,TABLE_COMMENT from information_schema.tables where TABLE_SCHEMA = 'test' order by UPDATE_TIME desc;

select TABLE_SCHEMA,TABLE_NAME,CREATE_TIME,UPDATE_TIME,TABLE_COMMENT from information_schema.tables where TABLE_SCHEMA = 'test' order by CREATE_TIME desc;


Friday, 11 September 2020

Python 3: OCI REST API - Set Annotation On Storage Array

In the NetApp OCI API Samples for Python (available from the OCI Operational Server WebUI > REST API documentation > API samples) there is a set_annotation.py which can be used to set the annotation on a specific volume. With a slight modification of the Python, we get a set_annotation_storage.py which can be used to set the annotation on a specific storage. An example of using it>

python set_annotation_storage.py --url https://YOUR_OCI_SERVER --user YOUR_REST_USER --password YOUR_PASSWORD --storage STORAGE_NAME -annotation SOME_ANNOTATION --value ANNOTATION_VALUE

Image: Insight REST API documentation > API samples

The Script

Save as say ‘set_annotation_storage.py’.

#!/usr/bin/env python 

"""

Set annotation for a specific storage

"""


from __future__ import print_function

import json

from oci_rest import OciRest, configure_command_line_parser


def find_storage(oci, storage_name):

  for storage in oci.get('assets/storages'):

    if storage['name'] == storage_name:

      return storage['self']

  return None


def annotate_storage(oci, storage_url, annotation_name, annotation_value):

  return oci.put('{}/annotations'.format(storage_url), data=json.dumps(

    [

      {

        "rawValue": annotation_value,

        "definition": {"name": annotation_name}

      },

    ]

  ))


if __name__ == "__main__":

  # Get the default command-line arguments (url, user, password)

  parser = configure_command_line_parser(usage='Set storage annotation')

  

  # Add additional arguments to configure the search criteria

  parser.add_argument('--storage', required=True, help="Storage name")

  parser.add_argument('--annotation', required=True, help="Annotation name (do NOT use label!)")

  parser.add_argument('--value', required=True, help="Annotation value to set")

  

  options = parser.parse_args()

  url, user, password = options.url, options.user, options.password

  

  with OciRest(url, user, password) as oci:

    storage_url = find_storage(oci, storage_name=options.storage)

    

    if storage_url is not None:

      new_annotation = annotate_storage(oci,

          storage_url,

          annotation_name=options.annotation,

          annotation_value=options.value)

      

      print('Created new annotation:\n', json.dumps(new_annotation))

      

    else:

      print("Could not find storage {}".format(options.storage))

APPENDIX: List of the 14 out-of-the-box ‘NetApp OCI API Samples for Python’

  • basic.py - print list of all storagePools with storage name and total allocatedCapacity
  • annotation_type.py - manipulate annotation types
  • business_entity.py - manipulate business entities
  • collection_report.py - Print status report for each datasource
  • create_user.py - Create OCI user
  • filter_storagepool_by_thresholds.py - print list of storage pools filtered by storage tier, and thresholds
  • filter_vms_by_storage.py - print virtual machines with dataStores on specific storage
  • license.py - Programmatically manipulate OCI license
  • import_patch.py - Import datasource patch
  • set_annotation.py - Set annotation for a specific volume
  • set_annotation_based_on_path.py - Set datacenter annotation for hosts based on their connected storage annotation
  • set_annotation_bulk.py - Set annotation for a group of volumes identified with prefix
  • set_ldap_config.py - set values for LDAP configuration
  • integrations.py - create, delete agents and integrations. Ingest integration data.