I’m having problems importing a large dataset from SAS

Welcome to the forum for MLwiN users. Feel free to post your question about MLwiN software here. The Centre for Multilevel Modelling take no responsibility for the accuracy of these posts, we are unable to monitor them closely. Do go ahead and post your question and thank you in advance if you find the time to post any answers!

Remember to check out our extensive software FAQs which may answer your question: http://www.bristol.ac.uk/cmm/software/s ... port-faqs/
Post Reply
MLwiN-User
Posts: 19
Joined: Thu Sep 03, 2009 1:39 pm

I’m having problems importing a large dataset from SAS

Post by MLwiN-User »

I'm trying to get a 2.5 million dataset from SAS into MLWiN. I've tried to use a SAS macro and also, to just export my data to a text file. When I use the macro to try to get my data into MLwiN, it says that it's "scanning data" -- it slows down at about 600,000 and stops at approximately 740,000. It doesn't stop at the same place every time! (for example, once it stopped at 733,800 and once at 759,700). I tried to increase the worksheet size to 50,000 and when I did, it stopped at 642,600. An error message pops up in a window: EXE file has encountered a problem and needs to close. We are sorry for the inconvenience. When I just try to import the text file, the same error message pops up. Wondering if you have any thoughts?!
MLwiN-Support
Posts: 16
Joined: Tue Oct 13, 2009 9:36 am

Re: I’m having problems importing a large dataset from SAS

Post by MLwiN-Support »

I would suggest importing a subset of your data, eg, a random sub sample of your higher level units. Even if you manage to get all the data in all but the simplest models would take a very long time to converge. A side benefit of using a sub-sample of data will be that you can check that you haven't overfit your model to your sub-sample of data by testing the final model on alternative sub-samples of the data. When model building you will want a considerably small sub-sample of your data, say 25,000 if the model is not too complex you can then estimate your final model on say 250,000.
danrodrics
Posts: 1
Joined: Sat Feb 12, 2011 5:23 am

Re: I’m having problems importing a large dataset from SAS

Post by danrodrics »

Thanks for posting this answer. However, I found the exact same question and answer combination in the FAQ section of this website. Is this coincidence or somebody just copied and pasted the relevant FAQ section in this thread?
CMM
Site Admin
Posts: 46
Joined: Fri Aug 28, 2009 10:56 am

Re: I’m having problems importing a large dataset from SAS

Post by CMM »

You are right about this. When we first created this forum in 2009 I copied a couple of genuine enquiries we had received and the answers we had given, to start the forum off, I haven't added any more since then though.
Post Reply