System Center Configuration Manager TechCenter >
System Center Configuration Manager Forums
>
Configuration Manager Internet Clients and Native Mode
>
IBCM only works for one day at a time, then has to be put back on the internal network
IBCM only works for one day at a time, then has to be put back on the internal network
- Yesterday I thought I made a breakthrough. I successfully got a laptop to download advertised software via the internet. I then pulled out half of the ram and a bit after the computer came back up ran a report to check on the installed ram amount. It changed! Then I installed some Windows Updates from a deployment that was advertised to the collection. Also good! Then today happened.
I configured an update deployment to hit the laptop late in the night and left the laptop up and running. I checked in the morning and it never ran. I tested a software deployment. The laptop was able to see the list of software advertised to it, but it would fail to download when I told it to install any piece of software. After a reboot the computer came back up with the same problem. The ram test now failed.
I pulled the laptop from the collection for the update deployment and reconnected it to our internal network and rebooted. Now it can download advertised software. I took it off of the network again and rebooted once more. It is still able to download software. I have now added the laptop back in to the collection for the update deployment (while the laptop is fully off of the internal network), and it worked. So this seems to be nailed down to some kind of time based problem, as multiple reboots off network don't cause a failure.
My setup: A single SCCM server for a site in native mode which is working fine with all computers on the intranet. We also have a primary child site in mixed mode for a separate forest, but I don't see any relevance for this problem. My guess is that this is in some way a cert issue. To that end, the internal and external server name(fqdn) is exactly the same, and I checked and was told I didn't need a SAN specified.
Laptop setup: Dell Latitude D520 running a Verizon wireless card with all other networking disabled. (nic not plugged in)
There are errors in many of the client logs, but I'm not sure which would be relevant for the problem, as many of these errors actually show up on intranet computers as well. I'll post any logs asked of me.
Thanks,
Aaron Barlow
Answers
- As a matter of fact, yes! It's a bit too soon to be sure, but we might be up and running on the test laptop.
The first registry file change didn't seem to make a huge difference, except that after a couple of days software downloads wouldn't fail instantly, but seemed to actually search before failing, even after a reboot (before it would search until a reboot, and then instant fail). However, if I ran the logon simulation script (below I've included the code. Add to a file, extension .js and use cscript to run), if I would wait a couple of minutes the software distribution would work again, but this would need to be run every other day or so. Not ideal.
On Friday the tech sent me another set of .reg files to run. I ran the files and had to shut down the laptop for the weekend. Brought it back up today, connected the Verizon card, and downloads are still flowing. On my end I plan on waiting for a couple of days to make sure that the connection is still good. I also added an advertisement on my site server, and advertised it to the client. The laptop saw the nee program and alerted me!
If this continues to work, the only thing left is figuring out why the registry files are required, and hopefully this is found to be something that MS can update the software for so that we don't need to run the .reg files on every computer that will use IBCM. Still, as long as it works once they are applied I think we can deal with it. I have my fingers crossed.
Start script:
//******************************************************************************
//
// Error
//
//******************************************************************************
function Error(sMsg, uExitCode)
{
WScript.Echo(sMsg);
WScript.Quit(uExitCode);
}
//******************************************************************************
//
// Main
//
//******************************************************************************
function Main()
{
//
// Check that we're running in a command prompt
//
if (WScript.FullName.toLowerCase().indexOf("wscript.exe") > 0)
{
Error("This script can only be run with cscript", 1);
}
try
{
var oUI,i;
i = 0;
oUI = WScript.CreateObject("SENS Logon Events");
do
{
oUI.StartShell("user");
//oUI.Logon("user");
i = i + 1
WScript.Echo("Simulated user logon " + i);
}
while (i < 1);
}
catch(e)
{
Error("Failed to invoke ACTION" + e.description + e.number);
}
WScript.Echo("Completed performing actions");
}
Main();
Script ended on line above.
Thanks,
Aaron- Marked As Answer byAaron Barlow Thursday, September 24, 2009 2:54 PM
- I'll check next week to see if we have any different laptops to check with, but everything we use for the most part (on a workstation level) is Dell. Multiple models, though I think for the most part they use the same networking software. I think one of the laptops I used was an Inspiron and another was a Latitude (not sure which exact models offhand). One interesting thing on further testing is that it appears as if the registry tweaks are not strictly needed, and the biggest thing is to make sure to add the external ip info for our management point into the hosts file. On the last laptop I tested I didn't tweak anything and after a couple of days the problem cropped up. I could go to the default website though using the browser, and I could hit the cdp as well. Pinging also was trying to hit the external address. Yet for all of this, I added the external IP into the hosts file, and downloading immediately started working.
- Marked As Answer byAaron Barlow Thursday, September 24, 2009 2:54 PM
All Replies
- Do you have the CDP published and accessible on the Internet to the Internet based client(s)?
Jason | http://myitforum.com/cs2/blogs/jsandys | http://blogs.catapultsystems.com/jsandys/default.aspx | Twitter @JasonSandys - I agree with Jason - this sounds suspiciously like a CRL issue. It's a very typical symptom that things work on the Internet for a while and then stop working when the CRL expires (by default, after 26 hours) and cannot be refreshed. Look for WINHTTP_CALLBACK_STATUS_FLAG_CERT_REV_FAILED in log files to help identify that this is the cause.
This blog posting might help if you need to publish the CRL so that it's accessible to your clients on the Internet, and how to confirm CRL access: http://blogs.technet.com/configmgrteam/archive/2009/05/01/how-to-publish-the-crl-on-a-separate-web-server.aspx
This blog posting explains how CRL issues manifest themselves in these scenarios: http://blogs.technet.com/wemd_ua_-_sms_writing_team/archive/2008/01/17/tips-tricks-hints-for-native-mode-and-internet-based-client-management-part-3-of-3.aspx
- Carol
This posting is provided “AS IS” with no warranties and confers no rights - Ok, I tested and we did not have the CDP published to the internet. I went ahead and published on a separate server using the first link that Carol posted. A couple of things though, firstly a question with the instructions.
"
To specify the separate Web server as a CDP
- On the CA server, load Certification Authority, right-click your CA, select Properties , and then click the Extensions tab.
- Ensure that CRL Distribution Point (CDP) is selected, and then click Add .
- In the Add Location dialog box, type the following and then click OK : http://<FQDN_of_Web_Server/<CRL_directory_name> For example, if your Web server was called server2.contoso.com and the virtual directory you created in IIS was called CRL, you would type http:// server2.contoso.com/crl
- Ensure that the following options are selected for this new entry:
- Include in CRLs. Clients use this to find Delta CRL locations.
- Include in the CDP extension of issued certificates
- Click OK . If you are prompted to restart Active Directory Certificate Services, click Yes .
Because of step 3, on any new certs I get a listing of http://servername.companyname.com/crl now. I also get the original listing which goes all the way to the .crl file. That difference seems odd to me, and that get intensified by:
"
To confirm new certificates contain new CDP
-
Request and issue a new certificate after you have completed the procedure "To specify the CRL on a separate Web server".
- On the requesting computer, load the Certificates MMC and locate the newly installed certificate.
- Double-click the certificate to view its properties.
- Click the Details tab and click the field CRL Distribution Points .
- View the values in this field. There will be multiple CRL distribution points listed so scroll down until you see the HTTP CRL distribution point that you added (for example: URL=http://server2.contoso.com/crl/.crl )
On step five it shows that the point should go all the way to the .crl file and yet it doesn't, again, because of the area above, it only lists to the folder container. I did perform the"To automatically publish the CRL on a separate server"
steps where I have listed the location all the way to the .crl file, but according to the instructions, that was not where I was to specificy "Include in CRLs." or "Include in the CDP extension..." so it doesn't show up on the client cert itself.
I am planning on taking the computer off network for a couple of days and watching to see if it fails again, but I just wanted to make sure that I have this done right.
Lastly, having anyone being able to download a CRL doesn't create any kind of security risk does it? I tested with someone completely off network that isn't part of the domain and he was able to download the CRL without any authentication.
Thanks,
Aaron - Aaron, I'm not sure I understand your concerns about the instructions. However, I did test this myself and as the blog posting said, they came from and were verified by Microsoft PKI experts in the field, so give them a go. A couple of other things that occurred to me:
These instructions are for a full CRL and doesn't include deltas. If you want to use delta CRLs and can't work out yourself how to configure this for this separate server environment, shoot me an email via SMSDocs@Microsoft.com - I've got some instructions from our field guys for this but I haven't had time to test it myself and write it up on the blog. But I'll share what I've got with you if you're interested.
Another option here is to bring in some PKI expertise because this is a complex area that could make or break your deployment - Microsoft Consulting Services or equivalent.
If configuring access to an external CRL is proving problematic, another option is to disable CRL checking on clients. With my Microsoft security hat on, I can't recommend that course of action but it is another option open to you, and I know some customers that run with this configuration. You can find more information about the pros/cons of enabling or disabling CRL checking on clients here: http://technet.microsoft.com/en-us/library/bb680463.aspx
CRLs cannot be accessed over HTTPS - so yes, on the Internet that means unauthenticated access and the data is unprotected in transit. The risk is no more and no less than when your browser uses CRL checking and you connect to online stores and give them your credit card details, or access your banking services online and connect over HTTPS. Before establishing that SSL connection, the browser checks the validity of the certificate on the server, and that includes checking whether the server certificate or any CAs in the chain have been revoked. To do that, it makes an HTTP connection to download the CRL for the CA that issued the certificate. I can't say whether all browsers do CRL checking by default, but I know that the latest IE browsers do. If you feel that this risk is higher than not using a CRL for your Internet-based clients, you do have the option of disabling it on the clients. But security best practice is to check the CRL before establishing a PKI connection.
- Carol
This posting is provided “AS IS” with no warranties and confers no rights - To rephrase my question with the instructions, in the "confirm new certificates contain new CDP" section, you have the new http address extending all the way through the .crl file under "CRL Distribution Points (on the cert)", however, per the "to specify a separate web server as a CDP" section, the http address will only go to the directory level under "CRL distribution points", which is how it looked when everything was done for me at first. I wasn't sure if that would suffice or not. Anyway, I have it done and am in the testing phase. I'll let you know whether or not I continue to have issues, and thanks to you both for the advice thusfar.
Aaron
EDIT: Everything looks great now as far as the CDP being added to the cert. I can see it in the CDP list on the cert both from the client machine and from the CA. I can get to the location and download the CRL via the Verizon card. I'm having a different problem now though. At this point If I boot the computer and it's hooked into the LAN, all is good, and I can switch to the Verizon card and download software. If I don't boot up on the LAN, however and first connect to the Verizon card, I get "The program is currently not available." I continue to get the message, even if I connect to the lan and run every action/wait an hour or two. So now it seems I have to be on the LAN first when I boot up, regardless of anything else.
EDIT 2: Ok, so that behavior isn't being consistent. For two or three times the first edit version of things was how it was going, but now it does actually seem to be working from a boot up completely not hooked up to the net, and then just turning on the Verizon card, and I cannot reproduce the earlier behavior. Really odd. Anyway, that quirk aside, I'm going to give it through Monday, and if it still works and hasn't been on the LAN, I'll call this resolved.- Edited byAaron Barlow Thursday, August 06, 2009 7:24 PM
- Alright, well, it appears my original problem is still present. After rebooting multiple times and being able to download advertised programs each time, I shut down the test computer for the night. Brought it back up and it is currently exhibiting the behavior of not being able to download programs. It did not take anywhere near 26 hours, probably more around 20 hours from when the cert was first issued until I tested and had the fail. It might have failed at some point last night earlier as well, so the failure happened somewhere in the 8-20 hour range from issue. Again, I can download the crl from the second .cdp location that is listed on the extensions tab of the cert while on the internet. At this point I'm not sure that's really what is causing the problem.
- Aaron - let's go the process of elimination route here to confirm whether it is a CRL-related issue or not. Assuming your client is domain-joined and the site is publishing to Active Directory, disable CRL checking on the site Site Mode tab of the Site properties), bring the client back onto the intranet and use this procedure to confirm that the client is configured to not check the CRL: http://technet.microsoft.com/en-us/library/bb932141.aspx
Only after confirming the Https flag value is 31 or 95, then test it again on the Internet.
If you're uncomfortable disabling CRL checking for all clients on the intranet for this testing period, reinstall your test client as Internet-only (http://technet.microsoft.com/en-us/library/bb694283.aspx), with the option /native: (no CRL checking).
The timing factor to bear in mind with CRLs is the validity period of the CRL from when the CA publishes it, and not when certificates are issued or when the client moves to the Internet. Is your CA configured to use delta CRLs?
- Carol
This posting is provided “AS IS” with no warranties and confers no rights - I have changed the site mode and am waiting for the flag to update (currently reads as 63). Interesting on the timing. I had all certs re-issued, and the problem happened at under 24 hours though, so I figured I'd be alright on the time. Good point on the delta CRLs. We are set to use them at a 1 day publication interval and have CRL interval at 1 week. Does that sound OK?
- The publication intervals sound fine, but if you're publishing delta CRLs the instructions you followed for your external CDP is for a base CRL only. So if the certificates have a delta CRL associated with them, clients on the Internet will not be able to access it - and they treat failure to download delta CRLs the same as they treat failure to access base CRLs. This goes back to my comment I made earlier "If you want to use delta CRLs and can't work out yourself how to configure this for this separate server environment, shoot me an email via SMSDocs@Microsoft.com - I've got some instructions from our field guys for this but I haven't had time to test it myself and write it up on the blog. But I'll share what I've got with you if you're interested."
We're busy trying to get SP2 RC out at the moment, so I haven't had time to test this yet - and remember, all this is external to Configuration Manager so the recommended advice about getting PKI advice still stands. As the blog post says, my product group does not specialize in PKI although we do what we can to help other customers. Perhaps by the time you've completed your CRL disabled testing I will have had time to test this myself.
On the timing issue, it comes down to when the clients last downloaded the CRL and the remaining validity period on the base and delta CRL. So if it's a CRL access problem, you could see it any time within the publication interval but never greater than that time. To give an example for your scenario, let's say a client on the intranet has been working fine on the intranet - it's downloaded the base CRL and has been downloading deltas. The last delta it downloaded was for the previous day and it has 3 hours remaining on the validity period. It moves onto the Internet and downloads policy, connecting to the Internet-based management point. The client is configured for CRL checking, so it checks the CRL for the server certificate on the Internet-based management point - the base CRL is still valid and the delta CRL is still valid, so there is no need to download newer ones. Then 6 hours later, it attempts to download policy again. This time the base CRL is still valid but the delta CRL has expired, so it tries to download the latest delta CRL and will fail if it's not accessible from the Internet.
Reissuing the certs doesn't change the validity period of the CRLs and even manually publishing the CRL outside the publication interval isn't a good timing test if the client has previously downloaded the CRL.
- Carol
This posting is provided “AS IS” with no warranties and confers no rights- Edited byCarol BaileyMSFT, ModeratorTuesday, August 11, 2009 1:45 PMMy terminology was off - correct term for initial CRL is a "base CRL", not "full CRL".
- OK, I think I've found what was missing for the delta CRLs - it's not enough to select the options "Publish Delta CRLs to this location" and "Clients use this to find Delta CRL locations". You also need the environment variable <DeltaCRLAllowed> in the path for both publication and the CDP. So it becomes file://\\<server>\share\<DeltaCRLAllowed>.crl and http://<FQDN>/<DeltaCRLAllowed>.crl When you do this, you see the +.crl file published as well as the .crl file - and this is the delta CRL. I've updated the blog posting with this information.
- Carol
This posting is provided “AS IS” with no warranties and confers no rights - Thank you for the update. Sorry about not catching the Delta point earlier. I slightly misread your post, and I noticed that some boxes were checked for delta access already in your earlier blog instructions so I thought for the purposes of what we were doing I'd be alright. I'm glad you now have the full instructions for getting deltas working.
On another note, yesterday afternoon I pulled the test laptop from the internal network after verifying the flag change, and it is still currently able to run advertised programs, so it looks like we probably have it (since as I had above, it always seemed to fail overnight). It hasn't technically been long enough to say for sure, so I'll test another time or two today, and then test tomorrow morning before we can reasonably be sure. Assuming for that, I'll follow the updated delta instructions.
Thanks,
Aaron - Yes, take your time testing this just to be sure. I also initially thought that checking the options to publish delta CRLs would be sufficient, which is why I had to do some additional investigation when I realized it wasn't enough. A lot of the documentation around Certificate Services include the use of variables, which I find confusing when I'm trying to work out the basics - so I always strip them down to bare minimum to confirm functionality before adding variables for customization. But in this case, the <DeltaCRLAllowed> variable is actually required. Then while I was investigating the other variables, I realized that the <CaName> and <CRLNameSuffix> should be used as well, so you'll see these also added to the instructions.
As I think I've said before, this side of PKI design is complex and I appreciate it's not easy to find the right information you need to be successful.
- Carol
This posting is provided “AS IS” with no warranties and confers no rights - Ok, bad news. Yesterday afternoon I attempted a download of a software application. It hung there for a few minutes searching for the download, and then gave the Cannot Run Program error. After rebooting it now immediately fails the download. Again, CRL checking is turned off for the site, and via Resource Explorer, the Https State Flags is set to 31. I'm not sure if there is a cert problem or something else, but does this fully eliminate CRLs as the (current) issue?
That's a surprise - at the moment I can't think what else is likely to be cause of something to stop working after a period of time. Yes, if you've disabled CRL checking on the client (and confirmed) and it's failing, then it can't be due to the client failing to access the CRL. Just to confirm - are your Internet-based site systems on the intranet and remain on the intranet? These also do CRL checking but if they are on the intranet they should always be able to access the default CDPs.
Testing a software distribution download is a good test, but to narrow the scope, can the client download policy when it's on the Internet? You can use Policy Spy from the toolkit to determine this. If you haven't used this before, it's pretty obvious if policy is failing to download but if in doubt, compare results against a working computer on the intranet. It would be useful to know if it's all communication failing, or something specific to software distribution.
- Carol
This posting is provided “AS IS” with no warranties and confers no rights- My internet site system is indeed on the intranet. It has two nics, one with an internet address and one for internal communication. Using Policy Spy, I can make an assignment request, but I never get the even of "CCM_PolicyAgent_AssignmentsReceived" on the laptop, though I do on my workstation when I make the request, so I'm assuming it's failing, although it never throws any kind of not received error.
Again though, I can get to the external cdp which is on that site system (just for testing), so I know that box is still available from the internet. - What error, if any are you getting in ccmexec.log when you request policy? If you enable verbose logging on the client, are there any detailed errors about the communication between the client and MP? Check out http://blogs.msdn.com/ameltzer/archive/2008/04/14/common-native-mode-client-mp-error-messages-and-what-to-do-about-them.aspx for more information on some of the errors and what they mean. Have you tried connecting to the MP through IE to make sure there's no certificate issues?
If all else fails, you may want to try using winhttptracecfg.exe (http://msdn.microsoft.com/en-us/library/aa384119%28VS.85%29.aspx ) to try to collect more details about what's happening under the covers with the SSL connection between the client and MP. That'll give you an obscene amount of detail about what's happening and may provide the key to figuring out what's going on if the ccmexec.log isn't helpful. - I am not currently getting any errors in the ccmexec.log when I request the policy. A few hours ago I was getting two errors: "Failed in WinHttpSendRequest API, ErrorCode=0x2efd" and "[CCMHTTP]HTTP ERROR: URL=http://cmut01.mpifix.com/ccm_system/request, Port=443, Protocol=https, SSLOptions=31, Code=12029, Text=ERROR_WINHTTP_CANNOT_CONNECT". It looks like all of your errors on your blog are regarding callback errors, so I don't think this applies. Anyway, the only way I know of to enable verbose logging is by re-running the installer. If I run "CCMSetup.exe CCMLOGLEVEL=0" as I think would be required, would I need to hook the test computer back up to the internal network? If so, it will be a while before I start getting errors again. What URL should I be hitting after the FQDN to test MP communication in IE?
Lastly, I can run the winhttptracecfg tool, but I'm unsure of the parameters I should be using. - Aaron, neither Adam nor I can work out what could be causing this timing-related issue and without any other suggestions coming in, I think we've reached the end of the scope that forum support can offer. At this point I recommend that you open a case with CSS or equivalent.
However, to answer your question about how to check the URL for the Internet-based management, you will need to export a client certificate with the private key and then import it into the browser, and then try connecting to https://<ServerName>/sms_mp/.sms_aut?mplist and https://<ServerName>/sms_mp/.sms_aut?mpcert, where <ServerName> is the FQDN for the Internet-based management point computer. You can find more information about this troubleshooting technique here: http://technet.microsoft.com/en-us/library/bb932118.aspx. If it is certificate-related, the SSLDiag.exe tool might be useful, but like WinHttpTraceCfg.exe, this is a tool that is external to Configuration Manager.
- Carol
This posting is provided “AS IS” with no warranties and confers no rights - Alright, I went ahead and called support. Currently the support guy has me trying a registry adjustment on the client. If it still has problems I am to run a script that emulates a login event. I'll update this with the details of any actual fix, in case somebody else has my woes. Thanks for trying, guys.
- Aaron, any update that you can share yet?
- As a matter of fact, yes! It's a bit too soon to be sure, but we might be up and running on the test laptop.
The first registry file change didn't seem to make a huge difference, except that after a couple of days software downloads wouldn't fail instantly, but seemed to actually search before failing, even after a reboot (before it would search until a reboot, and then instant fail). However, if I ran the logon simulation script (below I've included the code. Add to a file, extension .js and use cscript to run), if I would wait a couple of minutes the software distribution would work again, but this would need to be run every other day or so. Not ideal.
On Friday the tech sent me another set of .reg files to run. I ran the files and had to shut down the laptop for the weekend. Brought it back up today, connected the Verizon card, and downloads are still flowing. On my end I plan on waiting for a couple of days to make sure that the connection is still good. I also added an advertisement on my site server, and advertised it to the client. The laptop saw the nee program and alerted me!
If this continues to work, the only thing left is figuring out why the registry files are required, and hopefully this is found to be something that MS can update the software for so that we don't need to run the .reg files on every computer that will use IBCM. Still, as long as it works once they are applied I think we can deal with it. I have my fingers crossed.
Start script:
//******************************************************************************
//
// Error
//
//******************************************************************************
function Error(sMsg, uExitCode)
{
WScript.Echo(sMsg);
WScript.Quit(uExitCode);
}
//******************************************************************************
//
// Main
//
//******************************************************************************
function Main()
{
//
// Check that we're running in a command prompt
//
if (WScript.FullName.toLowerCase().indexOf("wscript.exe") > 0)
{
Error("This script can only be run with cscript", 1);
}
try
{
var oUI,i;
i = 0;
oUI = WScript.CreateObject("SENS Logon Events");
do
{
oUI.StartShell("user");
//oUI.Logon("user");
i = i + 1
WScript.Echo("Simulated user logon " + i);
}
while (i < 1);
}
catch(e)
{
Error("Failed to invoke ACTION" + e.description + e.number);
}
WScript.Echo("Completed performing actions");
}
Main();
Script ended on line above.
Thanks,
Aaron- Marked As Answer byAaron Barlow Thursday, September 24, 2009 2:54 PM
- It looks like we have this resolved to our satisfaction. In the end we had to configure software deployments not to require the user to be logged in to be able to stop using the above script. For some reason some other modifications were also needed due to changes that dell makes to the registry for network items on their laptops. I have everything 100% up on a test laptop and am testing this on other computers. Next week we have a bunch of laptop users coming in, and we'll configure SCCM on all of their systems. Hopefully this works for all of them.
Assuming next week goes off without huge issues, I'll come back and mark this as answered.
Thanks again,
Aaron - Aaron, from what I understand, this issue is specific to Dell laptops. Do you have any different laptops to test this with to confirm whether it's vendor-specific? And are all your Dell laptops the same model?
- Carol
This posting is provided “AS IS” with no warranties and confers no rights - I'll check next week to see if we have any different laptops to check with, but everything we use for the most part (on a workstation level) is Dell. Multiple models, though I think for the most part they use the same networking software. I think one of the laptops I used was an Inspiron and another was a Latitude (not sure which exact models offhand). One interesting thing on further testing is that it appears as if the registry tweaks are not strictly needed, and the biggest thing is to make sure to add the external ip info for our management point into the hosts file. On the last laptop I tested I didn't tweak anything and after a couple of days the problem cropped up. I could go to the default website though using the browser, and I could hit the cdp as well. Pinging also was trying to hit the external address. Yet for all of this, I added the external IP into the hosts file, and downloading immediately started working.
- Marked As Answer byAaron Barlow Thursday, September 24, 2009 2:54 PM
- Sorry it took so long. It's been an extremely busy couple of weeks. Anyway, outside of some toughbooks that I can't get ahold of easily, I don't have a non dell choice. It looks like we are not currently having problems since we added the reg/hosts fixes. May or not be too early to say in all cases, but I think this has stayed open long enough. Thanks for your help.
