none
Question on managing session - cookies using Powershell? RRS feed

  • Question

  • I would like to know on whether it is possible to manage a session using Powershell:

    1) Create a session and hold this session on www.finance.yahoo.com

    2) Do some coding here without closing the session

    3) Closing the session at the end

    Does anyone have any suggestions?
    Thanks in advance for any suggestions


    Thanks in advance for any suggestions

    Sunday, May 21, 2017 3:19 AM

Answers

All replies

  • I have used the following url to fetch the historical data from yahoo finance. From last 16th May, 2017 the url is not working.

    Seems like they have changed the url and the new url is:

    https://query1.finance.yahoo.com/v7/finance/download/AAL?period1=1494873000&period2=1494959400&interval=1d&events=history&crumb=l0aEtuOKocj

    In the above changed URL has a session cookie which is crumb. Is there any idea how to get this cookie programmatically(in PowerShell)?

    Does anyone have any suggestions?
    Thanks in advance for any suggestions :>

    Thanks in advance for any suggestions


    Saturday, May 20, 2017 10:35 PM
  • Sorry, but your query is outside the scope of this forum.

    -- Bill Stewart [Bill_Stewart]

    Sunday, May 21, 2017 12:41 PM
    Moderator
  • invoke-webrequest has a -session parameter, which specifies the session variable to use moving forward.

    This would mean any subsequent request using invoke-webrequest, can be used with the -session $name variable...e.g.

    invoke-webrequest -uri www.google.com.au -session sessionvar

    <do some stuff>

    invoke-webrequest -uri www.google.com.au -session $sessionvar

    Monday, May 22, 2017 3:27 AM
  • I would like to know on what difference is between invoke-webrequest and Enter-PSSession.  If I want to open a session for a period of time until manually close it, should I use invoke-webrequest or Enter-PSSession? If I use invoke-webrequest, would this code be one-time only and close it automatically when it is done?

    Do you have any suggestions?
    Thanks, to everyone very much for any suggestions :>

    Question :

    invoke-webrequest -uri www.google.com.au -session sessionvar, does it open session without "$"?

    invoke-webrequest -uri www.google.com.au -session $sessionvar, does it close session with "$"?


    Start Session
    Enter-PSSession - ComputerName SRV1

    Remove any opened session
    Remove-PSSession -id 5

    End Session
    Exit-PSSession


    Thanks in advance for any suggestions

    Monday, May 22, 2017 6:13 AM
  • The two commands are completely unrelated.  They cannot do what you are asking.

    Start by reading the help. After reading it rethink your question.

    help enter-possession -full

    help Invoke-WebRequest -full.

    Always read the help before asking a question.  It will help you ask a correctly stated question and will often give you the answer you seek.

    First ask what is a session?  There are remote sessions, web session and jam sessions as well many other kinds of sessions.  Decide which one you are asking about before reading the help.

    Invoke-WebRequest has two session parameters.  You need to understand how to use both.

    Enter-PsSession does not have or use cookies.


    \_(ツ)_/

    Monday, May 22, 2017 6:21 AM
  • Invoke-WebRequest has two session parameters.  You need to understand how to use both.

    Enter-PsSession does not have or use cookies.


    \_(ツ)_/

    Would Invoke-WebRequest be the right approach to solve my situation?

    Invoke-WebRequest has SessionVariable and WebSession parameters

    I need to manually create the session variable, then I need to use the -WebSession parameter instead of the -SessionVariable paramter, would it be correct?

    Do you have any example on how to use WebSession?
    Thanks, to everyone very much for any suggestions :>

    WebSession
    -WebSession takes an existing WebRequestSession object.

    SessionVariable
    -SessionVariable takes a string argument that is used to create a WebRequestSession object with that name.


    Thanks in advance for any suggestions



    Monday, May 22, 2017 7:50 AM
  • After a session is opened, I would like to know on when the right code is for downloading a csv file, I have tried following 2 approaches without luck.

    $URLresult = https://query1.finance.yahoo.com/v7/finance/download/%5EGSPC?period1=1492896439&period2=1495488439&interval=1d&events=history&crumb=kK.4Q7f/hsM

    Do you have any suggestions?
    Thanks, to everyone very much for any suggestions :>

    $url = "https://finance.yahoo.com/quote/%5EGSPC/history?p=%5EGSPC"
    $webrequest = Invoke-WebRequest -Uri $url -SessionVariable $Y

    1) (Invoke-WebRequest -Uri $URLresult -WebSession $Y -UseBasicParsing).Content | Out-File -FilePath D:\Yahoo.csv

    2) Invoke-WebRequest -Uri $URLresult -WebSession $Y | Out-File -FilePath D:\Yahoo.csv


    Thanks in advance for any suggestions




    Monday, May 22, 2017 9:44 PM
  • The object does not have cookies or the web site does not use session cookies.

    PS>$webrequest.headers['set-cookie']
    B=ac72pl5ci6nub&b=3&s=ke; expires=Tue, 22-May-2018 22:01:47 GMT; path=/; domain=.yahoo.com
    PS>


    \_(ツ)_/


    • Edited by jrv Monday, May 22, 2017 10:06 PM
    Monday, May 22, 2017 10:04 PM
  • If the object does not have cookies or the web site does not use session cookies.

    Could you please describe more on what object does not have cookies? and what following code do?

    PS>$webrequest.headers['set-cookie']
    B=ac72pl5ci6nub&b=3&s=ke; expires=Tue, 22-May-2018 22:01:47 GMT; path=/; domain=.yahoo.com
    PS>

    Do you have any suggestions?
    Thanks, to everyone very much for any suggestions :>


    Thanks in advance for any suggestions


    Tuesday, May 23, 2017 3:10 PM
  • # first request
    $webrequest = Invoke-WebRequest $url -SessionVariable websession
    # second request with same session Invoke-WebRequest $url -WebSession $websession $websession.Cookies.GetCookies($url)


    Please red the help carefully.

    Also look up all returned objects.

    https://msdn.microsoft.com/en-us/library/microsoft.powershell.commands.webrequestsession(v=vs.85).aspx


    \_(ツ)_/

    • Marked as answer by oemMicrosoft Wednesday, May 24, 2017 2:57 AM
    Tuesday, May 23, 2017 4:39 PM
  • Do you have any suggestions?

    Yes, I have a suggestion. You must start at the beginning and learn PowerShell basics.

    Elementary school students don't start at algebra. They start at basic mathematics and move up from there. Same concept here.

    The purpose of this forum is to answer specific scripting questions, not to teach beginner basics a tiny bit at a time. There are lots of good free PowerShell learning resources available. I would also recommend taking classes if possible.


    -- Bill Stewart [Bill_Stewart]

    Tuesday, May 23, 2017 4:45 PM
    Moderator
  • Thanks, to everyone very much for suggestions :>

    Thanks in advance for any suggestions

    Wednesday, May 24, 2017 2:57 AM
  • Referring to following coding, when I manually download file from Yahoo by clicking the download link, the data shows on different cells properly, but when I download using following coding, data is shown everything on the same single cell 1 without splitting into different columnas shown below:

    Date,Open,High,Low,Close,Adj Close,Volume
    1990-01-08,3842.000000,3942.000000,3742.000000,3850.000000,3850.000000,0

    Do you have any suggestions on what wrong is on following download script?
    Thanks, to everyone very much for any suggestions :>

    $URL = "https://query1.finance.yahoo.com/v7/finance/download/%5EGSPC?period1=1493072773&period2=1495664773&interval=1d&events=history&crumb=kK.4Q7f/hsM"

    $file = "C:\GSCP.csv"

    (Invoke-WebRequest -Uri $URL -WebSession $Y -UseBasicParsing).Content | Out-File -FilePath $file


    Thanks in advance for any suggestions



    Wednesday, May 24, 2017 10:30 PM
  • The web site is designed to prevent you from automating the file download. You must use the Yahoo finance API to download these tables.


    \_(ツ)_/


    • Edited by jrv Wednesday, May 24, 2017 10:47 PM
    Wednesday, May 24, 2017 10:46 PM
  • You can try this but it will likely not work.

    $url = 'https://query1.finance.yahoo.com/v7/finance/download/%5EGSPC?period1=1493072773&period2=1495664773&interval=1d&events=history&crumb=kK.4Q7f/hsM'
    $wc = New-Object System.Net.WebClient
    $wc.Credentials = Get-Credential yourname
    $wc.DownloadFile($url,'c:\GSCP.csv')
    
    


    \_(ツ)_/

    Wednesday, May 24, 2017 10:49 PM
  • Does Powershell have other function to download file instead of using (...).Content? which "(...).Content" should be the problem on downloading as a text.

    When I try following coding, the format is wrong neither.

    Invoke-WebRequest -Uri $URL -WebSession $Y | Out-File -FilePath $file

    Do you have any suggestions?
    Thanks, to everyone very much for any suggestions :>


    Thanks in advance for any suggestions


    Wednesday, May 24, 2017 10:50 PM
  • $url = 'https://query1.finance.yahoo.com/v7/finance/download/%5EGSPC?period1=1493072773&period2=1495664773&interval=1d&events=history&crumb=kK.4Q7f/hsM'
    $wc = New-Object System.Net.WebClient
    $wc.Credentials = Get-Credential yourname
    $wc.DownloadFile($url,'c:\GSCP.csv')

    An exception occurred during a WebClient request, which parameter not including session, so that should be the problem, does it have to use Invoke-WebRequest in this situation? because it requires a session.

    Do you have any suggestions?
    Thanks, to everyone very much for any suggestions :>


    Thanks in advance for any suggestions



    Wednesday, May 24, 2017 11:06 PM
  • The URL you have is not a file URL.  t is a trigger that causes a push download that triggers IE to ask if yu want to save the file.  YOu cannot bypass this with any PowerShell code.

    Post on the Yahoo developers forum for the location of the APIs that allow you to download files.

    Start here: http://meumobi.github.io/stocks%20apis/2016/03/13/get-realtime-stock-quotes-yahoo-finance-api.html


    \_(ツ)_/

    Wednesday, May 24, 2017 11:10 PM
  • The URL you have is not a file URL.  t is a trigger that causes a push download that triggers IE to ask if yu want to save the file.  YOu cannot bypass this with any PowerShell code.


    For Invoke-WebRequest, when URL triggers IE to ask for saving file, which already contains a filename if I manually download it.  I would like to know on how to not specific $file and use the any given triggered filename to save the file. Such as when I manually save file, ^GSPC is already shown a in pop up dialogue box, so how to output file based on given filename?

    Do you have any suggestions?
    Thanks, to everyone very much for any suggestions :>

    Invoke-WebRequest -Uri $URL -WebSession $Y | Out-File -FilePath $file

    Thanks in advance for any suggestions


    Thursday, May 25, 2017 2:02 AM
  • Once more - you cannot do this.  The URL is not a file link.  It is a trigger.  Invoke-WebRequest is NOT IE.  It does not automatically download a file.  Look at "Content"  it is just HTML and it tells you nothing.


    \_(ツ)_/

    Thursday, May 25, 2017 2:05 AM
  • Referring to following coding, when I manually download file from Yahoo by clicking the download link, the data shows on different cells properly, but when I download using following coding, data is shown everything on the same single cell 1 without splitting into different columnas shown below:

    CODING:

    $URL = "https://query1.finance.yahoo.com/v7/finance/download/%5EGSPC?period1=1493072773&period2=1495664773&interval=1d&events=history&crumb=kK.4Q7f/hsM"

    $file = "C:\GSCP.csv"

    (Invoke-WebRequest -Uri $URL -WebSession $Y -UseBasicParsing).Content | Out-File -FilePath $file

    OUTPUT :

    Date,Open,High,Low,Close,Adj Close,Volume
    1990-01-08,3842.000000,3942.000000,3742.000000,3850.000000,3850.000000,0

    Does Powershell have other function to download file instead of using (...).Content? which "(...).Content" should be the problem on downloading as a text.

    When I try following coding, the format is wrong neither.

    Invoke-WebRequest -Uri $URL -WebSession $Y | Out-File -FilePath $file


    Does anyone have any suggestions?
    Thanks in advance for any suggestions


    Thanks in advance for any suggestions

    • Merged by jrv Thursday, May 25, 2017 2:37 AM DUPLICATE
    Thursday, May 25, 2017 2:26 AM
  • Please don't keep reposting the same question. It will not get you a better answer.

    You have to try and understand why the web site does not allow downloads and why a portal has an API.  The data is costly.  They do not want automated screen scraping or they will be in violation of the license agreement with the data provider.  The APIs restrict the frequency of downloads and allow the downloaders to be identified.  Why else would they have an API?

    Spend some time learning how the Internet and portals work.  It is not all absolutely free.

    I gave you the link to the API. Please review it as it will allow you a simple way to do what you are trying to do.

    This is all true of Yahoo, MSN,, Google and almost all other portal sites..

      Financial data is not free.  Somebody has to pay for it.


    \_(ツ)_/

    Thursday, May 25, 2017 2:43 AM
  • I would like to know on what the standard coding on handling error as shown below

    Do you have any suggestions?
    Thanks, to everyone very much for any suggestions :>

    try {
    ???

    $URL = "https://query1.finance.yahoo.com/v7/finance/download/%5EGSPC?period1=1493072773&period2=1495664773&interval=1d&events=history&crumb=kK.4Q7f/hsM"

    $file = "C:\GSCP.csv"

    (Invoke-WebRequest -Uri $URL -WebSession $Y -UseBasicParsing).Content | Out-File -FilePath $file

     }

     catch [System.Exception] {

    ???

     }


    Thanks in advance for any suggestions

    Thursday, May 25, 2017 1:50 PM
  • I would like to know on what the standard coding on handling error as shown below

    Do you have any suggestions?
    Thanks, to everyone very much for any suggestions :>

    try {
    ???

    $URL = "https://query1.finance.yahoo.com/v7/finance/download/%5EGSPC?period1=1493072773&period2=1495664773&interval=1d&events=history&crumb=kK.4Q7f/hsM"

    $file = "C:\GSCP.csv"

    (Invoke-WebRequest -Uri $URL -WebSession $Y -UseBasicParsing).Content | Out-File -FilePath $file

     }

     catch [System.Exception] {

    ???

     }


    Thanks in advance for any suggestions

    I recommend opening a new question as this is a different issue from the original question.

    Before asking a new question review the help for Try/Catch:

    help about_try_catch


    \_(ツ)_/

    Friday, May 26, 2017 9:31 AM