none
Microsoft Machine Learning Server service expired automatically after a few hours!!! RRS feed

  • Question

  • I built an ML Server service in Python according to https://docs.microsoft.com/en-us/machine-learning-server/operationalize/python/quickstart-deploy-python-web-service

    However, the ML Server Service expired after several hours, which is NOT a good way for commercial application.

    To solve this, I have to

    1. Delete the service
    2. Create the same service again

    Otherwise, there is an error when calling the service: ‘401 Client Error: Unauthorized for url’ (User Consume Error after a few hours).

    And before re-create, I must delete the service first, otherwise there is ‘409 Client Error: Conflict for url’ (re-deploy without deleting). This means that the <g class="gr_ gr_859 gr-alert gr_spell gr_inline_cards gr_disable_anim_appear ContextualSpelling ins-del multiReplace" data-gr-id="859" id="859">url</g> IS there but can NOT be used and expired after a few hours!!!!!!!!!!!!

    Thus, could you advise a way to keep the service up and running 7/24 hours without deleting and re-creating?

    e.g.: refresh the service every two hours and etc.

    Log:

    Server Expired Detail: (after about 4 hours, it expired)

    User Consume Error:
    res = service.analyseImage('https://.....jpg')

    401 Client Error: Unauthorized for url: http:/....:12800/api/AnalyseImageService/v1.0

    ---------------------------------------------------------------------------
    HTTPError                                 Traceback (most recent call last)
    C:\Program Files\Microsoft\ML Server\PYTHON_SERVER\lib\site-packages\azureml\common\http_client.py in __send(self, method, resource_path, data, json, headers, raw_response, **kwargs)
        265         try:
    --> 266             response.raise_for_status()
        267 

    C:\Program Files\Microsoft\ML Server\PYTHON_SERVER\lib\site-packages\requests\models.py in raise_for_status(self)
        934         if http_error_msg:
    --> 935             raise HTTPError(http_error_msg, response=self)
        936 

    HTTPError: 401 Client Error: Unauthorized for url: http://....:12800/api/AnalyseImageService/v1.0

    During handling of the above exception, another exception occurred:

    HttpException                             Traceback (most recent call last)
    <ipython-input-126-4366bdfc4b02> in <module>()
          5 #'https://www.gopeople.com.au/wp-content/uploads/2017/02/FreshFlowersCourierOrientation.jpg'
          6 #'https://curtisfamilydental.com/wp-content/uploads/2016/05/Multiple-people.jpg'
    ----> 7 res = service.analyseImage('https://.....jpg')
          8 
          9 # -- Pluck out the named output `answer` as defined during publishing and print --

    C:\Program Files\Microsoft\ML Server\PYTHON_SERVER\lib\site-packages\azureml\deploy\server\service.py in analyseImage(self, image_path)
        105                 props[key.get('name')] = args[key.get('name')]
        106 
    --> 107             return self._consume(props)
        108 
        109         # -- build consume function and attach --

    C:\Program Files\Microsoft\ML Server\PYTHON_SERVER\lib\site-packages\azureml\deploy\server\service.py in _consume(self, body)
        317 
        318         # POST /api/:name/:version
    --> 319         response = self._http_client.post(self._api, json=body).json()
        320 
        321         output_schema = self._service.get('outputParameterDefinitions', {})

    C:\Program Files\Microsoft\ML Server\PYTHON_SERVER\lib\site-packages\azureml\common\http_client.py in post(self, resource_path, data, json, headers, raw_response, **kwargs)
        191 
        192     def post(self, resource_path, data=None, json=None, headers=None, raw_response=False, **kwargs):
    --> 193         return self.__send('POST', resource_path, data=data, json=json, headers=headers, raw_response=raw_response, **kwargs)
        194 
        195     def put(self, resource_path, data=None, headers=None, raw_response=False, **kwargs):

    C:\Program Files\Microsoft\ML Server\PYTHON_SERVER\lib\site-packages\azureml\common\http_client.py in __send(self, method, resource_path, data, json, headers, raw_response, **kwargs)
        268         except (RequestException, ConnectionError, HTTPError) as e:
        269             log.error(e)
    --> 270             raise HttpException(resp=response)
        271 
        272         if not raw_response and self.accept == 'application/json':

    HttpException: (401)
    Reason: Unauthorized
    HTTP response headers: {'Server': 'Kestrel', 'x-ms-request-id': 'b86b3c93-8942-47e5-b609-179c593dab19', 'Content-Length': '0', 'Date': 'Fri, 09 Nov 2018 14:14:01 GMT', 'X-Content-Type-Options': 'nosniff', 'WWW-Authenticate': 'Bearer error="invalid_token", error_description="The token is expired

    deploy()/re-create error without deleting:

    client = DeployClient(HOST, use=MLServer, auth=context)

    service_name = 'AnalyseImageService'

    service = client.service(service_name)\
            .version('v1.0')\
            .code_fn(analyseImage)\
            .inputs(image_path=str)\
            .outputs(answer=str)\
            .models(model = model)\
            .description('Image Analyse Service')\
            .deploy()


    409 Client Error: Conflict for url: http://....:12800/services/AnalyseImageService/v1.0

    ---------------------------------------------------------------------------
    HTTPError                                 Traceback (most recent call last)
    C:\Program Files\Microsoft\ML Server\PYTHON_SERVER\lib\site-packages\azureml\common\http_client.py in __send(self, method, resource_path, data, json, headers, raw_response, **kwargs)
        265         try:
    --> 266             response.raise_for_status()
        267 

    C:\Program Files\Microsoft\ML Server\PYTHON_SERVER\lib\site-packages\requests\models.py in raise_for_status(self)
        934         if http_error_msg:
    --> 935             raise HTTPError(http_error_msg, response=self)
        936 

    HTTPError: 409 Client Error: Conflict for url: http:/....:12800/services/AnalyseImageService/v1.0

    During handling of the above exception, another exception occurred:

    HttpException                             Traceback (most recent call last)
    <ipython-input-125-22fb193f8435> in <module>()
         10             sort_keys=True, indent=4)
         11 
    ---> 12 service = client.service(service_name)        .version('v1.0')        .code_fn(analyseImage)        .inputs(image_path=str)        .outputs(answer=str)        .models(model = model)        .description('Permyssion Image Analyse Service')        .deploy()
         13 
         14 

    C:\Program Files\Microsoft\ML Server\PYTHON_SERVER\lib\site-packages\azureml\deploy\operationalization.py in deploy(self)
        471                   service `deployed`.
        472         """
    --> 473         return self._op.deploy_service(self._name, **self._defs)
        474 
        475     def redeploy(self, force=False):

    C:\Program Files\Microsoft\ML Server\PYTHON_SERVER\lib\site-packages\azureml\deploy\server\mlserver.py in deploy_service(self, name, **opts)
        295         log.debug('MLServer.deploy_service(): \n   %s\n   %s', name, opts)
        296 
    --> 297         self._service.create_service(name, **opts)
        298 
        299         return self.get_service(name, version=opts.get('version'))

    C:\Program Files\Microsoft\ML Server\PYTHON_SERVER\lib\site-packages\azureml\deploy\server\services.py in create_service(self, name, **kwargs)
         65             if version is not None:
         66                 response = self._api.publish_web_service_version(
    ---> 67                     name, version, pub_request)
         68             else:
         69                 response = self._api.publish_web_service(name, pub_request)

    C:\Program Files\Microsoft\ML Server\PYTHON_SERVER\lib\site-packages\azureml\deploy\server\apis\services_api.py in publish_web_service_version(self, name, version, publish_request, **kwargs)
        194         resource = '/services/' + name + '/' + version
        195 
    --> 196         return self._http_client.post(resource, json=body)
        197 
        198     def publish_realtime_service(self, name, **kwargs):

    C:\Program Files\Microsoft\ML Server\PYTHON_SERVER\lib\site-packages\azureml\common\http_client.py in post(self, resource_path, data, json, headers, raw_response, **kwargs)
        191 
        192     def post(self, resource_path, data=None, json=None, headers=None, raw_response=False, **kwargs):
    --> 193         return self.__send('POST', resource_path, data=data, json=json, headers=headers, raw_response=raw_response, **kwargs)
        194 
        195     def put(self, resource_path, data=None, headers=None, raw_response=False, **kwargs):

    C:\Program Files\Microsoft\ML Server\PYTHON_SERVER\lib\site-packages\azureml\common\http_client.py in __send(self, method, resource_path, data, json, headers, raw_response, **kwargs)
        268         except (RequestException, ConnectionError, HTTPError) as e:
        269             log.error(e)
    --> 270             raise HttpException(resp=response)
        271 
        272         if not raw_response and self.accept == 'application/json':

    HttpException: (409)
    Reason: Conflict
    HTTP response headers: {'Server': 'Kestrel', 'x-ms-request-id': '36455d16-f4e5-4fb7-9f64-a8de6d7c6810', 'Content-Length': '171', 'Content-Type': 'application/json', 'Date': 'Fri, 09 Nov 2018 14:13:45 GMT', 'X-Content-Type-Options': 'nosniff'}
    HTTP response body: {"Link":"https://go.microsoft.com/fwlink/?linkid=830136","Message":"A service with same name and version exists. Use Patch to Modify.","ExceptionType":"ConflictException"}

    Monday, November 12, 2018 4:30 PM

All replies

  • Hi MaryWang,

    sorry to hear you were having issues - are you still having problems with this?

    If so, please let me know what authorization type you are using.

    thanks, Jeroen

    Friday, January 4, 2019 6:06 PM
    Owner