none
ML Server Admin util hangs on web node start with HTTPS configured RRS feed

  • Question

  • Hi,

    When I run "az ml admin node start -w" for my web node with HTTPS and AAD configured, Admin Utility hangs, though the web node is actually started. Could be that checking if web node started is performed using incorrect URL.

    Kind regards,

    Oleh Khoma

    • Changed type Oleh Khoma Tuesday, May 1, 2018 8:08 AM
    Friday, April 20, 2018 12:59 PM

Answers

  • Do you have the "Host" parameter filled out in your appsettings.json?

    Kestrel": {
        "Port": 12800,
        "Host": "localhost",

    • Marked as answer by Oleh Khoma Tuesday, May 1, 2018 8:08 AM
    Monday, April 30, 2018 3:50 PM

All replies

  • What information is given when you run with a --debug

    az ml admin node start -w --debug 


    Friday, April 20, 2018 3:22 PM
  • Nothing useful I'm afraid:

    C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin>az ml admin node start -w --debug
    Command arguments: ['ml', 'admin', 'node', 'start', '-w', '--debug']
    Event: Cli.PreExecute []
    Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x034A6C48>, <function OutputProducer.on_global_arguments at 0x0352FC90>, <function CLIQuery.on_global_arguments at 0x035538E8>]
    Event: CommandInvoker.OnPreCommandTableCreate [<function CLILogging.remove_logger_flags at 0x034A6C90>]
    Installed command modules ['acr', 'acs', 'advisor', 'appservice', 'backup', 'batch', 'batchai', 'billing', 'cdn', 'cloud', 'cognitiveservices', 'configure', 'consumption', 'container', 'cosmosdb', 'dla', 'dls', 'eventgrid', 'extension', 'feedback', 'find', 'interactive', 'iot', 'keyvault', 'lab', 'monitor', 'network', 'profile', 'rdbms', 'redis', 'reservations', 'resource', 'role', 'servicefabric', 'sql', 'storage', 'vm']
    Loaded module 'acr' in 0.007 seconds.
    Loaded module 'acs' in 0.005 seconds.
    Loaded module 'advisor' in 0.003 seconds.
    Loaded module 'appservice' in 0.007 seconds.
    Loaded module 'backup' in 0.005 seconds.
    Event: CommandLoader.OnLoadCommandTable []
    Loaded module 'batch' in 0.016 seconds.
    Loaded module 'batchai' in 0.005 seconds.
    Loaded module 'billing' in 0.004 seconds.
    Event: CommandLoader.OnLoadCommandTable []
    Loaded module 'cdn' in 0.005 seconds.
    Loaded module 'cloud' in 0.295 seconds.
    Loaded module 'cognitiveservices' in 0.003 seconds.
    Loaded module 'configure' in 0.002 seconds.
    Loaded module 'consumption' in 0.005 seconds.
    Loaded module 'container' in 0.004 seconds.
    Loaded module 'cosmosdb' in 0.006 seconds.
    Loaded module 'dla' in 0.102 seconds.
    Loaded module 'dls' in 0.004 seconds.
    Loaded module 'eventgrid' in 0.005 seconds.
    Loaded module 'extension' in 0.002 seconds.
    Loaded module 'feedback' in 0.001 seconds.
    Loaded module 'find' in 0.002 seconds.
    Loaded module 'interactive' in 0.001 seconds.
    Loaded module 'iot' in 0.006 seconds.
    Loaded module 'keyvault' in 0.007 seconds.
    Loaded module 'lab' in 0.152 seconds.
    Loaded module 'monitor' in 0.007 seconds.
    Loaded module 'network' in 0.023 seconds.
    Loaded module 'profile' in 0.005 seconds.
    Loaded module 'rdbms' in 0.005 seconds.
    Loaded module 'redis' in 0.005 seconds.
    Loaded module 'reservations' in 0.004 seconds.
    Loaded module 'resource' in 0.008 seconds.
    Loaded module 'role' in 0.004 seconds.
    Loaded module 'servicefabric' in 0.004 seconds.
    Loaded module 'sql' in 0.007 seconds.
    Event: CommandLoader.OnLoadCommandTable []
    Loaded module 'storage' in 0.266 seconds.
    Loaded module 'vm' in 0.020 seconds.
    Loaded all modules in 1.012 seconds. (note: there's always an overhead with the first module loaded)
    Extensions directory: 'C:\Users\khomao-adm\.azure\cliextensions'
    Found 1 extensions: ['azure-ml-admin-cli']
    None
    mls-admin-profile base location: C:\Users\khomao-adm\.azure
    mls-admin-profile location: C:\Users\khomao-adm\.azure\mls-admin-profile.json
    Loaded extension 'azure-ml-admin-cli' in 0.081 seconds.
    Event: CommandInvoker.OnPostCommandTableCreate [<function add_id_parameters at 0x037697C8>]
    Event: CommandInvoker.OnCommandTableLoaded []
    Event: CommandInvoker.OnPreParseArgs [<function _documentdb_deprecate at 0x03FA79C0>]
    Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x0352FCD8>, <function CLIQuery.handle_query_parameter at 0x03553930>]
    {'webnode': True, 'computenode': None}
    attempting to read file C:\Users\khomao-adm\.azure\mls-admin-profile.json as utf-8-sig
    =======================================================
    Process command: "C:\Program Files\Microsoft\ML Server\R_SERVER\o16n\dotnet\dotnet" "C:\Program Files\Microsoft\ML Server\R_SERVER\o16n\Microsoft.MLServer.Utils.AdminUtil\Microsoft.MLServer.Utils.AdminUtil.dll" node start --webnode
    =======================================================
     \ Processing (55 seconds) ..


    Friday, April 20, 2018 3:35 PM
  • Ok, if I wait long enough I can see that it was trying to use https://localhost, while the HTTPS certificate is for different name.

    FAILURE! Web Node was unable to start.
    Could not reach node at https://localhost/. Check the startup logs for more information under \o16n\Microsoft.MLServer.Utils.AdminUtil\logs.
    
    Event: CommandInvoker.OnTransformResult [<function _resource_group_transform at 0x0377EB28>, <function _x509_from_base64_to_hex_transform at 0x0377EB70>]
    Event: CommandInvoker.OnFilterResult []
    Event: Cli.PostExecute []
    

    Friday, April 20, 2018 3:37 PM
  • Perhaps a new parameter can be added to config or command line to specify which url to use when running this kind of service requests? Or would it be an overkill?
    Friday, April 27, 2018 3:29 PM
  • Do you have the "Host" parameter filled out in your appsettings.json?

    Kestrel": {
        "Port": 12800,
        "Host": "localhost",

    • Marked as answer by Oleh Khoma Tuesday, May 1, 2018 8:08 AM
    Monday, April 30, 2018 3:50 PM
  • Thank you, adding this parameter solved the problem.

    Oleh Khoma

    Tuesday, May 1, 2018 8:00 AM