I’m having problems importing a large dataset from SAS

[MLwiN-User]

I'm trying to get a 2.5 million dataset from SAS into MLWiN. I've tried to use a SAS macro and also, to just export my data to a text file. When I use the macro to try to get my data into MLwiN, it says that it's "scanning data" -- it slows down at about 600,000 and stops at approximately 740,000. It doesn't stop at the same place every time! (for example, once it stopped at 733,800 and once at 759,700). I tried to increase the worksheet size to 50,000 and when I did, it stopped at 642,600. An error message pops up in a window: EXE file has encountered a problem and needs to close. We are sorry for the inconvenience. When I just try to import the text file, the same error message pops up. Wondering if you have any thoughts?!

[MLwiN-Support]

I would suggest importing a subset of your data, eg, a random sub sample of your higher level units. Even if you manage to get all the data in all but the simplest models would take a very long time to converge. A side benefit of using a sub-sample of data will be that you can check that you haven't overfit your model to your sub-sample of data by testing the final model on alternative sub-samples of the data. When model building you will want a considerably small sub-sample of your data, say 25,000 if the model is not too complex you can then estimate your final model on say 250,000.

[danrodrics]

Thanks for posting this answer. However, I found the exact same question and answer combination in the FAQ section of this website. Is this coincidence or somebody just copied and pasted the relevant FAQ section in this thread?

[CMM]

You are right about this. When we first created this forum in 2009 I copied a couple of genuine enquiries we had received and the answers we had given, to start the forum off, I haven't added any more since then though.