CI insight-migration.py - Lessons Learned

 *Last updated 2022.03.21.

I'm running the Python tool insights-migration.py from a Windows machine at a financial services customer. I thought I'd keep track of a few notes / lessons learned.

Note: This tool comes as part of a ***** PS engagement (the tool has no customer support.) The link for a NetApp PS person to get the tool is:

  • https://ocim.corp.******.com/******/insights-migration

I'm publishing my notes as an unlinked page so I can recollect these lessons learned (the information is still searchable if you know what you're looking it, but not promoted.)

Installing

From the Windows Command Prompt.

Steps: 1) Upgrade pip. 2) Install requirements.txt. 3) Review the help file.

pip install --upgrade pip
pip install -r requirements.txt
python insights-migration.py -h

Configuring

This creates an empty config.ini file.

python insights-migration.py config

The config.ini file is configured with the details of the OCI server and the CI Tenant.

[OCI]
url = https://[FQDN_OF_OCI_SERVER]/
user = [ENTER_OCI_USERNAME]
password = [ENTER_OCI_PASSWORD]
 
[CI]
url = https://[ENTER_CI_TENANT].cloudinsights.netapp.com
token = [VERY_LONG_API_KEY]

Lessons learned:

  1. If you're just exporting data from OCI, you don't the CI bit configured correctly (when exporting data from OCI the program does not treach out to CI.)
  2. If you're just migrating collected data to OCI, you don't need the OCI bit configured correctly - i.e. you can put in a dummy IP address (when migrating data to CI, the program does not contact the OCI server.)

Exporting Cognos Reports

If you need to use the insight-migration.py tool to process OCI Cognos reports before manually importing them to CI Cognos (Cognos has no API to allow import of reports), then you must first do an export of the required folder(s) using the IBM Cognos Administration tool.

Cognos > Manage > Administrative Console

IBM Cognos Administration > Content Administration > New Export

And follow the wizard to export just the folder(s) that you need. Make sure 'Encrypt the content of the archive' is unchecked.

Lesson learned:

  • Only export the reports you need (i.e. tidy up/move into other folders the Cognos Reports that are no longer actively in use.)

Running the Collect

python insights-migration.py collect --content ExportCustomReports.zip

The --content CognosExportFilename.zip is optional. You don't have to process the Cognos Content Store Export.

Collect will collect the - Rules, Annotations, Applications, Datasources, Dashboards, PerfPolicy, Queries and Reports

When the collect completes, you will have the folder structure:

data > collected > reports

  • In data > collected will be a number of JSON files
    • AnnotationDefinition.json
    • AnnotationValues.json
    • Applications.json
    • Dashboard.json
    • Datasources.json
    • PerformancePolicy.json
    • Query.json
    • Rule.json
    • SystemInfo.json
  • In data > collected > reports will be your semi-processed OCI Cognos Reports (XML file per report with same folder structure as you had in Cognos.)

Lessons learned:

  • The good thing about the collect acquiring JSON files, is that, you can collect these files and customize them to some extent (especially useful if you are consolidating from many OCI servers to one CI server) i.e.:
    • Say you don't want to consolidate a number of OCI servers into one CI, you can combine all the Datasources and bring them across in one bang.
    • If you want to skip certain things from migration, you can "zero" out the JSON file to do this (more later.)
Before Running the Migrate

Before running the migrate, a comment on things you an zero out.
  • I've put no against AnnotationDefinition and AnnotationValues, as these are tied together in some way. I had not success zero-ing them.
  • Similarly Query and Rule, these are similarly tied together (Rules using Query), I did not try to zero them.
  • Everything else we can zero out.
  • Applications take a long time to migrate into CI, so you definitely want to do some tidy up before importing all your applications (this say 10 seconds per application, and if you have 100'000 applications, that's 1'000'000 seconds or over 11 days!)

AnnotationDefinition.json - Can zero, edit file to []
    AnnotationValues.json - Can zero, edit file to {} 
        Applications.json - Can zero, edit file to []
           Dashboard.json - Can zero, edit file to []
         Datasources.json - Can zero, edit file to []
   PerformancePolicy.json - Can zero, edit file to []
               Query.json - Can zero, edit file to []
                Rule.json - Can zero, edit file to []

Note: All [] square bracket except AnnotationValues which is {}.

Running the Migrate - Dry Run

python insights-migration.py -D migrate

Running the Migrate

python insights-migration.py migrate

Lessons learned:

  • The migrated folder gets renewed every time you run a migrate (don't expect to save data in that folder.)
  • With applications excluded, this runs pretty fast (less than 5 minutes, even for a very large OCI instance)
  • The tool will delete stuff (like annotation values previously imported), so watch out for stuff like the below. If you are consolidating OCIs, a recommendation would be to create new rules before starting the migrate (which will lock certain annotation values) rather than reusing the old regional OCI instance rules (which may make no sense at a global level.)
    • Warning: Data may be removed from Cloud Insights (specifically annotation lists get overwritten)
    • Error saving annotation: The annotation values "SPAIN", "GERMANY", "SWITZERLAND" are used by some queries or dashboards, and can not be deleted or updated.
    • Also check in the CI Audit log, and you might see a lot of 'Annotation Value ... removed from annotation.'
  • I did encounter some errors running the migrate "Error saving annotation: Failed to obtain the lock to run the task! Consider increasing lock wait time."

Note: As you'll see in the APPENDIX, it does not automatically migrate everything. Manual import accommodations are made for:

  • Manually add Annotation Rules, refer to the "data\migrated\Rules.csv" file.
  • Manually add Monitors from Performance Policies, refer to the "data\migrated\PerformancePolicy.csv" file.
  • Manually import Dashboards found in the "data\migrated\Dashboards.zip" file.
  • Manually fix the reports listed in the "data\migrated\ReportInvalid.csv" file.
  • Manually import Reports found in the "data\migrated\Reports.zip" file after fixing the invalid reports.
  • Once all Data collectors have successfully acquired data:
    • Manually import Application assignments found in "data\migrated\ApplicationImport.csv"
    • Manually import Annotation values found in "data\migrated\AnnotationImport.csv"

Overall Lesson Learned

This overall lesson learned applies if you are consolidating multiple OCI instances down to one CI instance. Really, I think you need to approach your migration in these steps:

  1. Run a collect with all Cognos Reports.
  2. Run all the other collects without Cognos Reports (since there is only one OCI Cognos Server.)
  3. Consolidate the collected data (this is something I've yet to fully work out how to do, but):
    1. Consolidate AnnotationDefinition.json and AnnotationValues.json
    2. Consolidate Applications.json
    3. (Optional*) Consolidate Dashboard
    4. Consolidate Datasources.json
    5. (Optional*) Consolidate PerformancePolicy
    6. Consolidate Query.json and Rule.json
  4. Run one migrate with all the consolidated collected data!
  5. Apply the finishing steps:
    1. AnnotationImport.csv
    2. ApplicationImport.csv
    3. (Optional*) Dashboards.zip
    4. (Optional*) PerformancePolicy.csv
    5. Rules.csv
*Optional for me because my customer never really used these in OCI.

~~~~~


APPENDIX A: Example of Migrate - Dry Run

Note: In the below I am skipping Applications, Dashboards, Datasources, Performance Policies:

C:\Users\USER\AppData\Local\Programs\Python\Python39>python insights-migration.py -D migrate
Error: Collected data from OCI server "..." does not match config: "https://..."
Would you like to continue? [y|default:n]:y
Migrating collected data to Cloud Insights: https://TENANT.cloudinsights.netapp.com from "data\collected"
    Annotation "Service Level" already exists
    Annotation "Violation Severity" already exists
    Annotation "Switch Level" already exists
    Annotation "Software Lifecycle" already exists
    Annotation "Support Provider" already exists
    Annotation "Cost Code Type" already exists
    Annotation "Life Cycle State" already exists
Migrated Annotations (50)
Migrated Applications (0)
    Business Entities (0) already exist
Migrated Business Entities to Annotation EnumValues (0)
Migrated Datasources (0)
    Query "All Switches" already exists
    Query "All Storage" already exists
    Query "All Storage Nodes" already exists
    Query "All Storage Pools" already exists
    Query "7-mode Storage" already exists
    Query "Inactive Storage Arrays" already exists
    Query "CIFS Share Inventory" already exists
    Query "OverSizedVMs" already exists
    Query "Services on Qtrees" already exists
Migrated Queries (15)
Reports: Collected dir "data\collected\reports" does not exist; skipping

Next Steps:

  • Use the Collector bulk edit option to:
    • Set the correct Acquisition Unit
    • Set passwords
    • Review Collector settings for accurate configuration
    • Resume collection, Collectors are postponed for a day upon creation
      • https://TENANT.cloudinsights.netapp.com/web/#/admin/dataCollectors/installed
  • Manually add Annotation Rules, refer to the "data\migrated\Rules.csv" file
    • https://TENANT.cloudinsights.netapp.com/web/#/manage/rules
  • Manually add Monitors from Performance Policies, refer to the "data\migrated\PerformancePolicy.csv" file
    • https://TENANT.cloudinsights.netapp.com/web/#/alerts/monitors
  • Manually import Dashboards found in the "data\migrated\Dashboards.zip" file
    • https://TENANT.cloudinsights.netapp.com/web/#/dashboards
  • Manually fix the reports listed in the "data\migrated\ReportInvalid.csv" file
  • Manually import Reports found in the "data\migrated\Reports.zip" file after fixing the invalid reports
  • Once all Data collectors have successfully acquired data:
    • Manually import Application assignments found in "data\migrated\ApplicationImport.csv"
    • Manually import Annotation values found in "data\migrated\AnnotationImport.csv"
    • Use the supplied "ci-import-metadata.py" for Application and Annotation import
    • NOTE: Annotation values that are found in *any* rule are not included
Complete

APPENDIX A: Example of Migrate - Full Run

Note: It is not much different to the dry run.

C:\Users\USER\AppData\Local\Programs\Python\Python39>python insights-migration.py migrate
Warning: Data may be removed from Cloud Insights https://TENANT.cloudinsights.netapp.com
Would you like to continue? [y|default:n]:y
Error: Collected data from OCI server "..." does not match config: "https://anociserver"
Would you like to continue? [y|default:n]:y
Migrating collected data to Cloud Insights: https://TENANT.cloudinsights.netapp.com from "data\collected"
  Annotation "Service Level" already exists
  Error saving annotation: None
  Error saving annotation: The annotation values "SPAIN", "GERMANY", "SWITZERLAND" are used by some queries or dashboards, and can not be deleted or updated.
  Annotation "Violation Severity" already exists
  Annotation "Switch Level" already exists
  Error saving annotation: Cannot get property 'id' on null object
  Error saving annotation: Cannot get property 'id' on null object
  Error saving annotation: Cannot get property 'id' on null object
  Error saving annotation: Cannot get property 'id' on null object
  Error saving annotation: Cannot get property 'id' on null object
  Annotation "Hardware Lifecycle" already exists
  Annotation "Software Lifecycle" already exists
  Annotation "Cost Code Type" already exists
  Annotation "Life Cycle State" already exists
Migrated Annotations (40)
Migrated Applications (0)
  Business Entities (0) already exist
Migrated Business Entities to Annotation EnumValues (0)
Migrated Datasources (0)
  Query "Symmetrix storage tier" already exists
  Query "Data Domain" already exists
  Query "All Storage" already exists
  Query "All Switches" already exists
  Query "All Storage Nodes" already exists
  Query "All Storage Pools" already exists
  Query "7-mode Storage" already exists
  Query "FlashSystem storage tier" already exists
  Query "AFF storage tier" already exists
  Query "Inactive Storage Arrays" already exists
  Query "CIFS Share Inventory" already exists
  Query "OverSizedVMs" already exists
  Query "Services on Qtrees" already exists
Migrated Queries (26)
Reports: Collected dir "data\collected\reports" does not exist; skipping

Next Steps:

- Use the Collector bulk edit option to:
  - Set the correct Acquisition Unit
  - Set passwords
  - Review Collector settings for accurate configuration
  - Resume collection, Collectors are postponed for a day upon creation
  https://TENANT.cloudinsights.netapp.com/web/#/admin/dataCollectors/installed

- Manually add Annotation Rules, refer to the "data\migrated\Rules.csv" file
  https://TENANT.cloudinsights.netapp.com/web/#/manage/rules

- Manually add Monitors from Performance Policies, refer to the "data\migrated\PerformancePolicy.csv" file
  https://TENANT.cloudinsights.netapp.com/web/#/alerts/monitors

- Manually import Dashboards found in the "data\migrated\Dashboards.zip" file
  https://TENANT.cloudinsights.netapp.com/web/#/dashboards

- Manually fix the reports listed in the "data\migrated\ReportInvalid.csv" file

- Manually import Reports found in the "data\migrated\Reports.zip" file after fixing the invalid reports

- Once all Data collectors have successfully acquired data:
  - Manually import Application assignments found in "data\migrated\ApplicationImport.csv"
  - Manually import Annotation values found in "data\migrated\AnnotationImport.csv"
  - Use the supplied "ci-import-metadata.py" for Application and Annotation import
  - NOTE: Annotation values that are found in *any* rule are not included

Complete

Comments

Post a Comment